pjfanning opened a new pull request, #3653: URL: https://github.com/apache/pekko/pull/3653
### Motivation `EventWriter` (internal, used to write events produced elsewhere into the journal) had a few gaps: - Events filtered out upstream leave sequence number gaps in the journal for a persistence id. - When a write failed and the highest sequence number lookup (used to detect duplicates) also failed, the writer threw and was restarted, leaving clients waiting until their ask timed out. - Snapshot events written through the writer could leave gaps before the snapshot sequence number. - The duplicate write failures that `EventWriter` relies on count towards the journal circuit breaker, which can then open and fail unrelated writes. ### Modification - Optional gap filling with `FilteredPayload` (`pekko.persistence.typed.event-writer.fill-sequence-number-gaps`, default `off`), with an LRU-bounded per-persistence-id cache of the latest sequence number (`latest-sequence-number-cache-capacity`, default `1000`). Filled events don't count towards the max batch size. - A failed highest sequence number lookup (`ReplayMessagesFailure` or ask failure) now fails the waiting writes with `"Journal write failed"` instead of restarting the writer. - `EventWriter.Write` gets `isSnapshotEvent`; gaps before a snapshot event are always filled. - `JournalProtocol.WriteMessages` gets `bypassCircuitBreaker` (default `false`), honoured by `AsyncWriteJournal`. `EventWriter` sets it, except after a failed sequence number lookup until the next successful write or lookup. - Tests: new `EventWriterFillGapsSpec` and `WriteMessagesSpec`, extended `EventWriterSpec`. ### Result `EventWriter` is more robust and can fill sequence number gaps. Default behaviour for normal persistent actors and journal plugins is unchanged. ### Tests Not run locally - left to CI. scalafmt and headerCreateAll were applied. ### References Ported from (Apache-2.0 since 2026-10-23): - https://github.com/akka/akka-core/pull/32095 - https://github.com/akka/akka-core/pull/32148 - https://github.com/akka/akka-core/pull/32191 - https://github.com/akka/akka-core/pull/32153 ### Notes for reviewers - **`JournalProtocol.WriteMessages` compat:** it is now a `final class` instead of a case class, with a 4th field `bypassCircuitBreaker` (default `false`). It keeps the 3-arg constructor, 3-arg `apply`, a 3-arg `copy` (with the same `copy$default$N` and keeping the flag), `_1`/`_2`/`_3` and the product methods for the first 3 fields, and equality/hashCode/toString (equality includes the flag). The companion extends version-specific parents in `persistence/src/main/scala-2` and `scala-3` (`WriteMessagesCompanion`). On Scala 2 that parent gives `AbstractFunction3` and `unapply(...): Option[(Seq, ActorRef, Int)]`. On Scala 3 it gives `Mirror.Product`, `fromProduct` and `unapply(...): WriteMessages`. This matches what the old case class companions compiled to, so pattern matches compiled against 2.0.0 still link. New code can still write `case WriteMessages(messages, ref, id)`. A toy program was compiled separately on 2.13 and 3.3 to check that a client compiled against the old case c lass runs against the new class. sbt MiMa was **not** run locally, so no MiMa excludes were added for the `persistence` module. If the `Check / Binary Compatibility` job reports something, add a targeted exclude. - **EventWriter internals** (`private[pekko]`, `@InternalStableApi`): - `EventWriterSettings` keeps a 2-arg constructor and `apply` (gap filling off). - `Write` keeps the 6-arg constructor and `apply` (not a snapshot event), and its companion still extends `AbstractFunction6`. - The changed `copy`/`unapply`/`copy$default$N`/`_N` of those two classes, and the private state and message classes, are excluded in `persistence-typed/src/main/mima-filters/2.0.x.backwards.excludes/eventwriter-fill-gaps.excludes` and `eventwriter-snapshot-event.excludes`. These excludes were written by hand without running MiMa, so CI may ask for adjustments. - **New config:** `fill-sequence-number-gaps = off` and `latest-sequence-number-cache-capacity = 1000` keep the current behaviour by default. - **Known follow-up candidate:** a failed sequence number lookup issued for gap filling or a snapshot event (not for a write failure) hits `MaxSeqNrForPidFailed` with an empty `waitingForReply`, and the writes queued in `waitingForSeqNrLookup` are not failed. Calling `head` on the empty sorted seq would also throw and restart the writer. This is unchanged from the ported behaviour. Similarly, after a lookup triggered by a snapshot event, all writes queued during that lookup are handled as snapshot events (so their gaps are filled too). - These changes are for Pekko 2.0.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
