HeartSaVioR commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#issuecomment-534586325 > The older instance, which is relinquished by all the threads is eventually closed. That will become the most complicated part when we allow concurrent usage of instance. If we don't allow concurrent usage (like Commons Pool) we are always safe to close the instance whenever it sits in pool. If not, reference counting comes up and additional thread safety follows. > Can we simplify - in other words, can we reduce the amount of code needed? I guess someone would concern about performance and cost between maintaining multiple producers vs concurrent using one producer, and then the concern would be valid, though we can't say former has much more overhead unless being measured. If we want simplicity, IMHO former would be much more simpler, especially we have already dealt with complexity on handling commons pool on consumer side. We just need to refactor the code to share the codebase of commons pool between consumer and producer. (And the PR does it nicely.)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
