Debbie,
In our case, the issue was with custom SSL implementation and we need to
fix a bug related to buffer over flow handling. This may not be applicable
to you, if you are using open source kafka. The best way to troubleshoot
this problem would be to take couple of thread dumps with few
intuit.com<mailto:deborah...@intuit.com>>,
"users@kafka.apache.org<mailto:users@kafka.apache.org>"
<users@kafka.apache.org<mailto:users@kafka.apache.org>>
Subject: Re: FW: Mirrormaker stops consuming
Debbie,
In our case, the issue was with custom SSL implem
Hi Raja,
We seem to be encountering the same problem you were seeing where our
producer thread becomes blocked for some reason.
We also see our producer queue is full,
and for some reason, the producer isn¹t pulling from the queue and sending
to our brokers.
We were wondering if you might be
Hi Raja,
We seem to be encountering the same problem you were seeing
Where our producer thread becomes blocked for some reason.
We were wondering if you might be able to share how you fixed your problem if
you fixed it.
Thanks,
Debbie
We recently upgraded to kafka 0.8.2.1 and found issues with mirrormaker
that randomly stops consuming. We had to restart the mirrormaker process to
resolve the problem. This problem has occurred several times in past two
weeks.
Here is what I found in analysis:
When this problem happens:
The issue is that multiple consumers feed into all the data channels.
So they will all eventually block if any data channel becomes full.
The mirror maker on trunk is significantly different so this is not an
issue on trunk.
On Fri, May 22, 2015 at 12:37:01PM -0400, Rajasekar Elango wrote:
Thanks for pointers Joel. Will look into SSLSocketChannel. Yes this was
working fine before upgrade.
If its just one producer thread stuck on write, it might affect only one
consumer thread/partition. But we found consuming stopped for all
topic/partitions. Or Is it only single data channel
The threaddump suggests that one of the producers
(mirrormaker-producer-6) is blocked on write for some reason. So the
data-channel for that producer (which sits between the consumers and
the producer) is full which blocks the consumers from progressing.
This appears to be in your (custom)
It is possible that the message produced rate is slower than the consumed
message rate which results in insufficient space left for the internal data
channel mirror maker uses to buffer the data from consumer to producer. You
can check histogram MirrorMaker-DataChannel-Size to see if any space
