sjvanrossum commented on PR #36029: URL: https://github.com/apache/beam/pull/36029#issuecomment-3243328020
> ## Summary of Changes > Hello @sjvanrossum, I'm Gemini Code Assist[1](#user-content-fn-1-ef9e0c3879beddb0afd344ee8b441586)! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! > > This pull request addresses a potential starvation issue in the KafkaIO connector's ReadFromKafkaDoFn where the processElement method could continuously poll for records without yielding to the Beam runner. The changes ensure that the polling loop respects the consumerPollingTimeout, allowing the Splittable DoFn to periodically yield control, thereby improving the responsiveness and stability of Kafka-consuming Beam pipelines. > > ### Highlights > * **Prevent indefinite polling**: The ReadFromKafkaDoFn's poll loop now exits after consumerPollingTimeout has elapsed, preventing it from running continuously without yielding to the runner, even when records are constantly arriving. > * **Improved yielding behavior**: By ensuring the poll loop terminates within the specified timeout, the SDF (Splittable DoFn) can yield control back to the Beam runner, improving overall pipeline stability and resource management. > * **Explicit ProcessContinuation**: The processElement method now explicitly returns ProcessContinuation.resume(), ensuring the runner is informed to continue processing the element. > > Using Gemini Code Assist > ## Footnotes > 1. Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487). [↩](#user-content-fnref-1-ef9e0c3879beddb0afd344ee8b441586) Good bot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
