bvolpato opened a new pull request, #40936: URL: https://github.com/apache/beam/pull/40936
`SpannerChangeStreamOrderedWithinKeyGloballyIT` set the flush timer for a key from the first record that arrived for that key. This assumes that the first record to arrive is the earliest, but Beam does not guarantee arrival order. When a record from a later batch arrives first, the timer is too late and one flush contains every batch. The test then fails because it expects one output per batch. The logs of failed GCP IO precommit runs show this directly. In [this run](https://github.com/apache/beam/actions/runs/41271524119/job/124745102618), the third batch started at `04:20:48.003` and the first at `04:20:25.714`. Keys 1 and 2 then logged `Setting timer at 2026-11-01T04:20:58.003Z`, which is the third batch plus the 10 second interval. [This run](https://github.com/apache/beam/actions/runs/40139252052/job/121007943827) shows the same pattern. This change moves the timer back when a record with an earlier commit timestamp arrives. A record that is not late holds the watermark back until it arrives, so the timer cannot fire too early. The test keeps its per-batch assertion. `SpannerChangeStreamOrderedByTimestampAndTransactionIdIT` has similar timer logic, but its assertion does not depend on the number of flushes, so I did not change it. To validate this without GCP, I replayed the commit timestamps from the failed run through the test's DoFns with `TestStream` on DirectRunner, in a scratch test that is not part of this PR. With the old logic, commit order passes and every out-of-order arrival tested fails with merged output like the CI failure. With this change, every arrival order tested produces the expected output. Spotless, Checkstyle, and test compilation ran locally. The integration test needs GCP, so it was not run locally. The GCP IO precommit on this PR runs it. <!-- Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). - [x] Replace these instructions with a description of your change and why you have made it. - [ ] Mention the appropriate issue in your description (for example: `addresses #123` or `fixes #456`). - [ ] Update `CHANGES.md` with noteworthy changes. See [CONTRIBUTING.md]() for more details. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
