pjfanning opened a new pull request, #487: URL: https://github.com/apache/pekko-persistence-r2dbc/pull/487
Port of [akka/akka-persistence-r2dbc#445](https://github.com/akka/akka-persistence-r2dbc/pull/445) (`fix: backtracking queries and filtered events`). The code is copied from the akka-persistence-r2dbc 1.1.2 release, which is available under the Apache License, Version 2.0. ## Problem Backtracking queries can stop too early. This can create a gap between the normal db queries and the backtracking queries that's longer than the backtracking window, and if a projection instance is then restarted, there are events that are completely missed, or rejected because of earlier missed events. The switch-from-backtracking check compares how many rows were returned by the past query to `bufferSize - 1`. The 1 is for the event at the initial timestamp, which has already been seen by the previous query and will be filtered from the result. If there are events that share timestamps (because they were emitted together), then there can be multiple events filtered out of the result set. In this case, it will switch from backtracking mode before catching up, and while it can still progress one query at a time, it can fall too far behind. ## Fix Keep track of the seen count for the latest timestamp when backtracking, and copy this over as the number of events expected to be filtered for the next query. Two fields are added to the internal `BySliceQuery.QueryState`: - `latestBacktrackingSeenCount` — how many rows shared the latest backtracking timestamp - `backtrackingExpectFiltered` — carried into the next query and used by `switchFromBacktracking` A debug log was also added for when the next offset falls outside the backtracking window. ## Notes - No new MiMa filters are needed: the existing `core/src/main/mima-filters/2.0.x.backwards.excludes/backtracking-improvements.excludes` already excludes the `QueryState` case class synthetics, and `core/mimaReportBinaryIssues` passes for both Scala 2.13 and Scala 3.3 with this change. - `EventsBySliceBacktrackingSpec` gets a new test (`predict backtracking filtered events based on latest seen counts`) and a small `query.buffer-size = 10` so the scenario is reachable. The spec needs a database, so it was not run locally — relying on CI for that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
