Tim, Thanks for looking into this !
>> it looked as if some of them use the same logic as the ruby client, and >> might also be affected. Could you please list the clients that you think might have the same bug ? > are you actually using any client besides the java / scala one in production > at linkedin? No. Thanks, Neha On Tue, Oct 25, 2011 at 9:16 AM, Tim Lossen <t...@lossen.de> wrote: > jun, > > the ruby client (maintained by alejandro crosa) is here: > > https://github.com/acrosa/kafka-rb > > i noticed just now that he also seems to work at linkedin? > > last commit on github is a bugfix, on october 14. but there > also seem to be some changes on the apache side which are not > in the github version ... maybe alejandro can best sort this > out himself. > > while looking over the other clients when we where hunting for > this bug, it looked as if some of them use the same logic as > the ruby client, and might also be affected. > > are you actually using any client besides the java / scala one > in production at linkedin? > > cheers > tim > > > On 2011-10-25, at 17:58 , Jun Rao wrote: > >> Tim, >> >> Thanks for the update. What's the github url for the ruby client? Has it >> diverged from what's in Apache? >> >> I agree with you that we should consider excluding those clients not well >> maintained from our distribution. >> >> Jun >> >> On Tue, Oct 25, 2011 at 7:46 AM, Tim Lossen <t...@lossen.de> wrote: >> >>> ok, we finally traced this issue to a bug in the ruby kafka client, >>> which we were able to fix -- the topic was never corrupted. >>> >>> (we sent a pull request to the maintainer of the client on github.) >>> >>> BTW, i do not think that it is a good idea to include an (outdated) >>> copy of the ruby client (and other clients) in the kafka distribution. >>> maybe better *link* to the actual client projects? >>> >>> cheers >>> tim >>> >>> On 2011-10-24, at 21:30 , Tim Lossen wrote: >>> >>>> ok, thanks -- tomorrow we'll try investigate further ... >>>> >>>> >>>> On 2011-10-24, at 9:12 PM, Neha Narkhede wrote: >>>> >>>>> Tim, >>>>> >>>>>> what if the CRC32 checksum is correct, but the internal binary message >>> structure is not? >>>>> >>>>> The CRC check involves recomputing the CRC and then checking against >>>>> the stored CRC in the header. The probability of that matching is >>>>> extremely low. >>>>> >>>>> Corruption is also possible if the broker crashes in the middle of a >>>>> flush. In that case, when the broker restarts, it detects an unclean >>>>> shutdown, runs recovery on the logs and truncates the log if the CRC >>>>> check fails at some message. >>>>> >>>>> Also, we compute the CRC only on the payload of the message. So >>>>> technically, some bits could get flipped in the header of the message. >>>>> >>>>> Thanks, >>>>> Neha >>>>> >>>>> On Mon, Oct 24, 2011 at 12:07 PM, Tim Lossen <t...@lossen.de> wrote: >>>>>> what if the CRC32 checksum is correct, but the internal binary message >>>>>> structure is not? >>>>>> >>>>>> >>>>>> On 2011-10-24, at 8:56 PM, Jay Kreps wrote: >>>>>> >>>>>>> It is not supposed to be possible. We include a CRC32 with each >>> message, >>>>>>> so >>>>>>> invalid requests should be detected and rejected. But that does not >>>>>>> preclude >>>>>>> the possibility that we missed a case. >>>>>>> >>>>>>> -Jay >>>>>>> >>>>>>> On Mon, Oct 24, 2011 at 11:41 AM, Tim Lossen <t...@lossen.de> wrote: >>>>>>> >>>>>>>> hi, >>>>>>>> >>>>>>>> is it possible for a faulty client to "corrupt" a topic on the >>> broker, >>>>>>>> so that consumers cannot consume messages any more? or does >>>>>>>> the broker protect itself against this? >>>>>>>> >>>>>>>> i am asking because we seem to have run into such a situation. >>>>>>>> we are using a perl producer and a ruby consumer. the per lib might >>>>>>>> be a bit outdated. >>>>>>>> >>>>>>>> cheers >>>>>>>> tim >>>>>>>> >>>>>>>> -- >>>>>>>> http://tim.lossen.de >>>>>>>> >>>>>>>> >>>>>> >>>>>> -- >>>>>> http://tim.lossen.de >>>>>> >>>>>> >>>> >>>> -- >>>> http://tim.lossen.de >>>> >>> >>> -- >>> http://tim.lossen.de >>> >>> >>> >>> > > -- > http://tim.lossen.de > > > >
