Hey Jörg, Unfortunately when the high level consumer hits a corrupt message, it enters an invalid state and closes. The only way around this is to iterate your offset by 1 in order to skip the corrupt message. This is currently not automated. You can catch this exception if you are using the simple consumer client, but unfortunately mirrormaker uses the high level client.
There have been some corrupt producer message bugs related to using snappy compression recently, but this does not seem to be the same as your problem. Does MM stop on the exact same message each time ( https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-ConsumerOffsetChecker )? I would suggest triple checking that your configurations are the same across all DC's (you mentioned that MM mirrors successfully to another DC with no problem), as well as examine the problem message to see if you can find anything different about it when compared to the others (See: https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-SimpleConsumerShell ). Your only other recourse is to iterate past the problem offset. On Thu, Oct 1, 2015 at 1:22 AM, Jörg Wagner <joerg.wagn...@1und1.de> wrote: > Hey everyone, > > I've been having some issues with corrupted messages and mirrormaker as I > wrote previously. Since there was no feedback, I want to ask a new question: > > Did you ever have corrupted messages in kafka? Did things break? How did > you recover or work around that? > > Thanks > Jörg >
