HeartSaVioR edited a comment on issue #25853: [SPARK-21869][SS] Apply Apache Commons Pool to Kafka producer URL: https://github.com/apache/spark/pull/25853#issuecomment-564030756 I just went through implementing new approach of SPARK-21869: please take a look at the branch below, which I revise the cache logic for producer. https://github.com/HeartSaVioR/spark/tree/SPARK-21869-new-cache-on-top-of-reverting-SPARK-21869 The actual diff is in this commit, https://github.com/HeartSaVioR/spark/commit/ae2b607089ee115f5993eacf26444c3266fef81f which is on top of revert commit of this PR. To minimize the change I just took the existing class and object but in other side trying to encapsulate, hence things may not seem to be beauty. (For example, I'd really like to avoid having only object and don't expose a chance to inject anything for testing - that's why I separated KafkaDataConsumer and pool implementations.) But it should be easier for us to review the concept. Please let me know if the concept/direction looks good; if we feel this as good alternative for SPARK-21869, I guess we can revert the commit first, and raise a PR for new approach afterwards.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
