vishalmore90 opened a new pull request, #40242: URL: https://github.com/apache/beam/pull/40242
Fixes #40240 **Context** A high volume of unprocessed or duplicate events entering the OrderedEventProcessor loop could previously trigger a `CommitTooLargeException` in Dataflow, because DLQ emissions were unconditionally appended to the bundle without respecting the `maxNumberOfResultsToProduce` pagination cap. **Changes** * Tracked duplicate output in `ProcessingState.java`. * Augmented `ProcessorDoFn.reachedMaxResultCountForBundle` to consider duplicate volume. * Refactored the `skipProcessing` block inside `ProcessorDoFn.processBufferedEventRange` to break gracefully on limits instead of continuing infinitely. * Added validation test `testLargeNumberOfDuplicatesPaginatesCorrectly`. **Checklist** - [x] Code style follows the repository guidelines - [x] Automated tests run and pass - [x] Appropriate comments and documentation updated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
