damccorm opened a new pull request, #41166: URL: https://github.com/apache/beam/pull/41166
### Summary Fixes intermittent row-count assertion failures in `BigQueryIOIT.testWriteThenRead` during `PerformanceTests BigQueryIO Streaming Java` ([run 42794252307](https://github.com/apache/beam/actions/runs/42794252307), [run 42300162313](https://github.com/apache/beam/actions/runs/42300162313)). ### Root Cause When `writeMethod` is `STREAMING_INSERTS`, `BigQueryIO.Write` uses `StreamWithDeDup`, which assigns random UUIDs in `TagWithUniqueIds` before `Reshuffle` and writes via BigQuery legacy `tabledata.insertAll` (which only provides best-effort at-least-once deduplication). When Dataflow Streaming retries a bundle upstream of `Reshuffle` or `insertAll` best-effort deduplication misses a small number of retried rows across 10,485,760 records, `testRead()` reads slightly more rows than `sourceOptions.numRecords` (e.g., `10485801` or `10485952` vs `10485760`) and fails `assertEquals`. ### Fix In `BigQueryIOIT.testRead()`, assert `actualRecords >= sourceOptions.numRecords` when `writeMethod` is `STREAMING_INSERTS`, while preserving exact `assertEquals(sourceOptions.numRecords, actualRecords)` for batch / exact write methods. ### Verification - Triggered `beam_PerformanceTests_BigQueryIO_Streaming_Java.yml` workflow run on branch: [`42840841274`](https://github.com/apache/beam/actions/runs/42840841274) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
