ok, we finally traced this issue to a bug in the ruby kafka client, which we were able to fix -- the topic was never corrupted.
(we sent a pull request to the maintainer of the client on github.) BTW, i do not think that it is a good idea to include an (outdated) copy of the ruby client (and other clients) in the kafka distribution. maybe better *link* to the actual client projects? cheers tim On 2011-10-24, at 21:30 , Tim Lossen wrote: > ok, thanks -- tomorrow we'll try investigate further ... > > > On 2011-10-24, at 9:12 PM, Neha Narkhede wrote: > >> Tim, >> >>> what if the CRC32 checksum is correct, but the internal binary message >>> structure is not? >> >> The CRC check involves recomputing the CRC and then checking against >> the stored CRC in the header. The probability of that matching is >> extremely low. >> >> Corruption is also possible if the broker crashes in the middle of a >> flush. In that case, when the broker restarts, it detects an unclean >> shutdown, runs recovery on the logs and truncates the log if the CRC >> check fails at some message. >> >> Also, we compute the CRC only on the payload of the message. So >> technically, some bits could get flipped in the header of the message. >> >> Thanks, >> Neha >> >> On Mon, Oct 24, 2011 at 12:07 PM, Tim Lossen <t...@lossen.de> wrote: >>> what if the CRC32 checksum is correct, but the internal binary message >>> structure is not? >>> >>> >>> On 2011-10-24, at 8:56 PM, Jay Kreps wrote: >>> >>>> It is not supposed to be possible. We include a CRC32 with each message, >>>> so >>>> invalid requests should be detected and rejected. But that does not >>>> preclude >>>> the possibility that we missed a case. >>>> >>>> -Jay >>>> >>>> On Mon, Oct 24, 2011 at 11:41 AM, Tim Lossen <t...@lossen.de> wrote: >>>> >>>>> hi, >>>>> >>>>> is it possible for a faulty client to "corrupt" a topic on the broker, >>>>> so that consumers cannot consume messages any more? or does >>>>> the broker protect itself against this? >>>>> >>>>> i am asking because we seem to have run into such a situation. >>>>> we are using a perl producer and a ruby consumer. the per lib might >>>>> be a bit outdated. >>>>> >>>>> cheers >>>>> tim >>>>> >>>>> -- >>>>> http://tim.lossen.de >>>>> >>>>> >>> >>> -- >>> http://tim.lossen.de >>> >>> > > -- > http://tim.lossen.de > -- http://tim.lossen.de
