Tim, Thanks for the update. What's the github url for the ruby client? Has it diverged from what's in Apache?
I agree with you that we should consider excluding those clients not well maintained from our distribution. Jun On Tue, Oct 25, 2011 at 7:46 AM, Tim Lossen <t...@lossen.de> wrote: > ok, we finally traced this issue to a bug in the ruby kafka client, > which we were able to fix -- the topic was never corrupted. > > (we sent a pull request to the maintainer of the client on github.) > > BTW, i do not think that it is a good idea to include an (outdated) > copy of the ruby client (and other clients) in the kafka distribution. > maybe better *link* to the actual client projects? > > cheers > tim > > On 2011-10-24, at 21:30 , Tim Lossen wrote: > > > ok, thanks -- tomorrow we'll try investigate further ... > > > > > > On 2011-10-24, at 9:12 PM, Neha Narkhede wrote: > > > >> Tim, > >> > >>> what if the CRC32 checksum is correct, but the internal binary message > structure is not? > >> > >> The CRC check involves recomputing the CRC and then checking against > >> the stored CRC in the header. The probability of that matching is > >> extremely low. > >> > >> Corruption is also possible if the broker crashes in the middle of a > >> flush. In that case, when the broker restarts, it detects an unclean > >> shutdown, runs recovery on the logs and truncates the log if the CRC > >> check fails at some message. > >> > >> Also, we compute the CRC only on the payload of the message. So > >> technically, some bits could get flipped in the header of the message. > >> > >> Thanks, > >> Neha > >> > >> On Mon, Oct 24, 2011 at 12:07 PM, Tim Lossen <t...@lossen.de> wrote: > >>> what if the CRC32 checksum is correct, but the internal binary message > >>> structure is not? > >>> > >>> > >>> On 2011-10-24, at 8:56 PM, Jay Kreps wrote: > >>> > >>>> It is not supposed to be possible. We include a CRC32 with each > message, > >>>> so > >>>> invalid requests should be detected and rejected. But that does not > >>>> preclude > >>>> the possibility that we missed a case. > >>>> > >>>> -Jay > >>>> > >>>> On Mon, Oct 24, 2011 at 11:41 AM, Tim Lossen <t...@lossen.de> wrote: > >>>> > >>>>> hi, > >>>>> > >>>>> is it possible for a faulty client to "corrupt" a topic on the > broker, > >>>>> so that consumers cannot consume messages any more? or does > >>>>> the broker protect itself against this? > >>>>> > >>>>> i am asking because we seem to have run into such a situation. > >>>>> we are using a perl producer and a ruby consumer. the per lib might > >>>>> be a bit outdated. > >>>>> > >>>>> cheers > >>>>> tim > >>>>> > >>>>> -- > >>>>> http://tim.lossen.de > >>>>> > >>>>> > >>> > >>> -- > >>> http://tim.lossen.de > >>> > >>> > > > > -- > > http://tim.lossen.de > > > > -- > http://tim.lossen.de > > > >
