pjfanning opened a new pull request, #665: URL: https://github.com/apache/pekko-projection/pull/665
### Motivation When the slice distribution of a projection is changed (for example downscaling from 2 to 1 projection instances), one of the previous instances may have been further behind than the backtracking window. Starting from the latest stored offset could then miss events from the slices that instance was responsible for. ### Modification - `R2dbcOffsetStore` reads `projection_key` together with the timestamp offset records. - When the records loaded for the slice range come from more than one projection key, `readOffset` starts from the latest offset of the earliest slice instead of the overall latest offset. - `getOffset` (used by the management API) is unchanged and still returns the latest offset. - Ported from akka/akka-projection#1012 (Akka Projection 1.5.0, now Apache 2.0 licensed). It is adapted to Pekko's `R2dbcOffsetStore`, which has the SQL inline instead of a separate DAO, so the change covers both the Postgres/Yugabyte and MySQL offset stores. There is no schema change: `projection_key` is an existing column. ### Result Projections no longer skip events after a change of slice distribution when the previous instances progressed unevenly. ### Tests - New `R2dbcTimestampOffsetStoreSpec` test "start from earliest slice when projection key is changed", run against local Postgres: fails without the fix (`...00:03:31Z was not equal to ...00:00:32Z`, i.e. resumes at the far-ahead offset), passes with it. - `sbt "r2dbc-int-test/testOnly"` for `R2dbcTimestampOffsetStoreSpec`, `R2dbcOffsetStoreSpec`, `R2dbcOffsetStoreStateSpec`, `R2dbcTimestampOffsetProjectionSpec`: pass. On the first run, one test failed: "skip failing events when using RecoveryStrategy.skip" (single projection key, unrelated). The spec passed twice on rerun. - `sbt "r2dbc/mimaReportBinaryIssues"` (vs 1.0.0): no issues - native scalafmt (diff against upstream/main); whitespace diff check clean - Not run locally: MySQL/Yugabyte integration tests (left to CI) ### References Refs akka/akka-projection#1012 (https://github.com/akka/akka-projection/pull/1012) - source of the ported code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
