pjfanning opened a new issue, #528: URL: https://github.com/apache/pekko-persistence-r2dbc/issues/528
#515 ports "snapshots as starting points" from the akka-persistence-r2dbc 1.2.0 release (akka/akka-persistence-r2dbc#410, #412, #438, #464, #469), the last Akka release that is under the Apache License 2.0. Akka changed the feature after 1.2.0, and those changes are not in #515. ### Fixed in #515 (Pekko code, nothing copied from Akka) Found in the review of #515. Both bugs were in the 1.2.0 code: - Events dropped by the snapshot filter did not move the query offset forward. If a whole page was dropped, `currentEventsBySlicesStartingFromSnapshots` completed too early and `eventsBySlicesStartingFromSnapshots` got stuck on that page. The filter now runs after the offset is updated. - With `NoOffset`, the event query started at the oldest snapshot timestamp, so older events of entities without a snapshot were never emitted. The event query now starts at the given offset. ### Not in #515 1. **Offsets go backwards (akka/akka-persistence-r2dbc#717, part 2).** All snapshot envelopes come first, then the events. An event after the snapshots can have an earlier timestamp than the snapshots already emitted. A projection that saves offsets and restarts before it catches up can resume from a point after events it has not processed yet. #515 only documents this. Akka 1.3.12 fixed it by rebuilding the query around `eventsBySlices`: it looks up an entity's snapshot sequence number the first time it sees the entity, emits the snapshot when it reaches that sequence number, and skips the events before it. That keeps the offsets in order. 2. **Batched snapshot lookups (akka/akka-persistence-r2dbc#759, 1.3.17).** A performance follow-up to #717: it batches the snapshot sequence number lookups instead of doing one database round trip per new entity. This only matters if we adopt the #717 design. 3. **Dropped events are still deserialized.** Since the #515 fix, the filter runs on envelopes, so events before a snapshot are deserialized before they are dropped. A design like #717 would avoid that. ### Licensing akka-persistence-r2dbc 1.3.12 (#717) is under the BSL with Change Date 2029-02-06, and 1.3.17 (#759) has a later Change Date. That code must not be copied or adapted before then. Any fix before 2029 needs a clean-room implementation, based on the problem descriptions and our own tests, not on the Akka code. ### Proposed work - A clean-room redesign that keeps the offsets in order (item 1), with a test that restarts from a saved offset halfway through and checks that no events are lost. - Then batch the lookups (item 2), which also covers item 3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
