gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-571139975
Thanks guys to step in and dealt with this! I think we've came up with a
better approach.
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-555409664
Well, the resource increase is definitely there which excludes option 1.
I agree that
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-554939486
@zsxwing Since this PR is a tradeoff it's questionable because not everybody
has the same threshold such cases.
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-554722904
This will be the first task on Monday. I've mentioned that documentation is
not covered and will come.
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-548740087
> Note that my goal of experiment is not proving the patch outperforms than
current.
Same understanding,
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-548511277
First of all thanks for your efforts @HeartSaVioR
I've just analyzed through the application what you've
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-546870975
@ScrapCodes @vanzin @zsxwing @tdas @koeninger may I ask for help here?
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-535970383
During cluster tests it has been turned out the producer lifecycle has a
problem. Going to check it next week...
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-535945541
Test type1: Endurance test
https://github.com/gaborgsomogyi/spark-structured-kafka-stress-app
That said
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-535464441
Adding when having results. Since this is critical path in the streaming
area I would like to keep the endurance
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-534923681
That said in my previous comment
[here](https://github.com/apache/spark/pull/25853#issuecomment-534519947)
Apache
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-534541196
> Can we simplify - in other words, can we reduce the amount of code needed?
The rough stand in terms of
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-534531458
> do we need more than one instance of kafka producer on the executor?
Not necessarily. It's a trade-off
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-534519947
> What happens to the old instance?
If none of the tasks are using the old instance, then after the eviction
gaborgsomogyi commented on issue #25853: [SPARK-21869][SS] Apply Apache Commons
Pool to Kafka producer
URL: https://github.com/apache/spark/pull/25853#issuecomment-533145143
cc @vanzin @HeartSaVioR since you've major knowledge about consumer caching
cc @ScrapCodes since you've filed the
15 matches
Mail list logo
