gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#issuecomment-555409664 Well, the resource increase is definitely there which excludes option 1. I agree that https://github.com/apache/spark/pull/19096 contains parts which can be brittle. Additionally no other ideas which is not horror complicated which conclude me on option 3. > IMO, writing zero messages in 10 minutes is usually not expected by the user and indicates some other issues. Thanks for sharing the initial issue, I agree it can happen rarely and the root cause must be fixed. Just a question here. The old code which creates single instance uses [expireAfterAccess](https://github.com/apache/spark/blob/dc2abe51ca2d3d702d6b6457301c3ca9c7244212/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaProducer.scala#L62) which means producer will be invalidated and closed when the cache item not accessed for 10 minutes. Do you think the original implementation will cover batch use-cases (since you've suggested `don't fix SPARK-21869`)?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
