Hmm, well CPU is pretty much zero. Heap is barely used. I even made the
task put method be a noop other than to log time-since-last call. No
change. With yourkit I see that ES has a thread that is sleeping, but it's
in a monitor thread pool and clearly not blocking kafka. Anyway, I even
removed
Barry,
It might help to know whether you're hitting a (single threaded) CPU limit
or if the bottleneck is elsewhere. Also, how large on average are the
messages you are consuming? There's nothing that'll force batching like
you're talking about. You can tweak any consumer settings via
I am running a connect consumer that receives JSON records and indexes into
elasticsearch. The consumer is pushing out 300 messages/s into the a topic
with a single partition. The connect job is configured with 1 task. (This
is all for testing).
What I see is that push is called about every 10s
