rahmjan opened a new pull request, #560: URL: https://github.com/apache/pekko-persistence-r2dbc/pull/560
### Motivation `skipPubSubTooFarAhead` drops a pub-sub event that is more than the backtracking window after the latest backtracking event. That position only moved when backtracking returned events, so after any quiet period longer than the window, and after a start, pub-sub events were dropped and only delivered later by the database query, one `refresh-interval` or more after they were written. ### Modification - When a backtracking round completes right after a normal query that found nothing and had no upper bound, everything before a known database time has been read: the normal query read everything after `latest` up to its database time minus `behind-current-time`, and the round everything before `latest` up to its database time minus `backtracking.behind-current-time`. - The database time is read before the round starts. The normal query's own database time is bounded by it minus the time passed since the normal query was created, measured with `System.nanoTime`, so no other database call is needed. - `liveBySlices` emits a progress marker envelope with that time before the next query's rows. It doesn't change the query state. - `skipPubSubTooFarAhead` uses the marker as the backtracking position and removes it from the stream. The markers are only emitted when events are published and backtracking is enabled, which is when that filter is in the stream. - The backtracking position no longer moves back on an older backtracking event, which a round re-reads after a marker. ### Result After an idle period, pub-sub events are accepted again once a backtracking round has completed, instead of being dropped until backtracking returns an event. In the test environment from the issue, 100% of the events of all five services came through pub-sub with this change, with a median delay of 15 to 24 milliseconds, and the integration test suites of the two services that wait for events between steps ran 1.9 and 6.7 times faster. ### Tests - `EventsBySlicePubSubBacktrackingProgressSpec` (new): after idling longer than the backtracking window, the next event arrives through pub-sub. It fails on main and passes with this change. - `EventsBySlicePubSubSpec`: new test of `skipPubSubTooFarAhead` with progress markers - `sbt "core/testOnly org.apache.pekko.persistence.r2dbc.query.* org.apache.pekko.persistence.r2dbc.state.*"`: MySQL 148 succeeded, PostgreSQL 152 succeeded - `sbt +core/Test/compile`: success on 2.13.18, 3.3.9, 3.9.0, 3.10.0 - `sbt core/scalafmtCheck core/Test/scalafmtCheck core/headerCheck core/Test/headerCheck`: success - `sbt +core/mimaReportBinaryIssues`: success; on main `mimaPreviousArtifacts` is empty, so nothing is compared - `sbt checkCodeStyle`: success - `sbt core/test` on PostgreSQL with all three changes of this series combined: 474 succeeded - Not run: YugabyteDB, MariaDB ### References Fixes #557, follows up #380 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
