[
https://issues.apache.org/jira/browse/SPARK-25151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-25151:
---------------------------------
Description:
KafkaDataConsumer contains its own logic for caching InternalKafkaConsumer
which looks like can be simplified via applying Apache Commons Pool. Benefits
of applying Apache Commons Pool are following:
* We can get rid of synchronization of KafkaDataConsumer object while
acquiring and returning InternalKafkaConsumer.
* We can extract the feature of object pool to outside of the class, so that
the behaviors of the pool can be tested easily. Now it doesn't have detailed
tests and only covers reported issues.
* We can get various statistics for the object pool, and also be able to
enable JMX for the pool.
> Apply Apache Commons Pool to KafkaDataConsumer
> ----------------------------------------------
>
> Key: SPARK-25151
> URL: https://issues.apache.org/jira/browse/SPARK-25151
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 2.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> KafkaDataConsumer contains its own logic for caching InternalKafkaConsumer
> which looks like can be simplified via applying Apache Commons Pool. Benefits
> of applying Apache Commons Pool are following:
> * We can get rid of synchronization of KafkaDataConsumer object while
> acquiring and returning InternalKafkaConsumer.
> * We can extract the feature of object pool to outside of the class, so that
> the behaviors of the pool can be tested easily. Now it doesn't have detailed
> tests and only covers reported issues.
> * We can get various statistics for the object pool, and also be able to
> enable JMX for the pool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
