Jungtaek Lim created SPARK-25151:
------------------------------------
Summary: Apply Apache Commons Pool to KafkaDataConsumer
Key: SPARK-25151
URL: https://issues.apache.org/jira/browse/SPARK-25151
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.4.0
Environment: KafkaDataConsumer contains its own logic for caching
InternalKafkaConsumer which looks like can be simplified via applying Apache
Commons Pool. Benefits of applying Apache Commons Pool are following:
* We can get rid of synchronization of KafkaDataConsumer object while acquiring
and returning InternalKafkaConsumer.
* We can extract the feature of object pool to outside of the class, so that
the behaviors of the pool can be tested easily. Now it doesn't have detailed
tests and only covers reported issues.
* We can get various statistics for the object pool, and also be able to enable
JMX for the pool.
Reporter: Jungtaek Lim
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
