Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/21685
In general `KafkaConsumer.poll` should take couple of seconds but 10+ is
extreme high. The question `why it takes so long?` has to be answered first. In
the processing time chart I see a trend which shows peeks periodically.
I have a couple of questions without checking the code:
* You're trying to commit something into 2.4 but in the test result I see
with 2.1.0 version. Have you tested it with 2.4? This part of the code has
significantly changed. Results with this version would be better.
* In the `before` case the input rate was approximately the same just like
in the `after` case constantly. After the initial good performance something
wrong happened and decreased the rate significantly. What happened exactly
there? Maybe memory filled up and not able to poll things without GC (just
guessing)?
* Have you considered/tested when driver/receiver dies? Guarantees are
quite important.
* Have you tested it with receivers? Some results would be excellent.
All in all IMO we haven't reached the root cause and because of that not
able to judge whether it's the right solution.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
