GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/spark/pull/22138
[SPARK-25151][SS] Apply Apache Commons Pool to KafkaDataConsumer
## What changes were proposed in this pull request?
KafkaDataConsumer contains its own logic for caching InternalKafkaConsumer
which looks like can be simplified via applying Apache Commons Pool. Benefits
of applying Apache Commons Pool are following:
* We can get rid of synchronization of KafkaDataConsumer object while
acquiring and returning InternalKafkaConsumer.
* We can extract the feature of object pool to outside of the class, so
that the behaviors of the pool can be tested easily.
* We can get various statistics for the object pool, and also be able to
enable JMX for the pool.
This patch brings additional dependency, Apache Commons Pool 2.6.0 into
`spark-sql-kafka-0-10` module.
## How was this patch tested?
Existing unit tests as well as new tests for object pool.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/spark SPARK-25151
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22138.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22138
----
commit c82f3064fa8744f91b5c8a92645588dc9d53ba35
Author: Jungtaek Lim <kabhwan@...>
Date: 2018-08-17T09:56:31Z
[SPARK-25151][SS] Apply Apache Commons Pool to KafkaDataConsumer
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
