bvolpato opened a new pull request, #27702: URL: https://github.com/apache/beam/pull/27702
It was recently identified that `ReadFromKafkaDoFn`'s `restrictionTracker` leaked connections that are not being polled anymore. This can cause considerable performance degradation for Kafka clusters, as they rely on the `connections.max.idle.ms` to clean it up. It was also hard to track where new connections were coming from, so I've added a few logs to indicate when a new connection is needed in the critical path. `KafkaLatestOffsetEstimator` doesn't hold any state, so by reusing it across backlog statistics for the same DoFn instance, we can save a huge amount of connections.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
