Thank you for confirming my suspicion. T#
On Sun, Mar 1, 2015 at 9:48 AM, Magnus Edenhill <mag...@edenhill.se> wrote: > Hi Theo, > > seems like you found the answer yourself, the server may return partial > messages. > While parsing a MessageSet you simply ignore any message whose length > > remainingBufferLength. > > Regards, > Magnus > > > 2015-03-01 8:55 GMT+01:00 Theo Hultberg <t...@iconara.net>: > > > That the message set size was always the same when this happened made me > > start looking for that number, and it turned out that was what I had set > as > > the MaxBytes field of the requests. This make me think that what happens > is > > that the next message to fetch is larger than this, but it's sent anyway. > > Is this correct? Is this how it is supposed to work? I'm looking through > > other clients to see how they deal with this, and I think I might have > > found the same thing in the Go client ( > > https://github.com/Shopify/sarama/blob/master/message_set.go#L72-L76), > and > > the corresponding comment in the protocol guide. Is what I have found the > > thing described as "As an optimization the server is allowed to return a > > partial message at the end of the message set. Clients should handle this > > case."? > > > > yours, > > Theo > > > > On Sat, Feb 28, 2015 at 10:19 PM, Theo Hultberg <t...@iconara.net> > wrote: > > > > > Here's an example of a frame that looks malformed, I've split the bytes > > > apart and annotated the pieces with the names from the Kafka protocol > > guide > > > ( > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse > > > ). > > > > > > Notice that the MessageSetSize is 0x00000800 (2048), but the > MessageSize > > > is 0x000010B3 (4275). The size prefix of the Value field agrees with > > > MessageSize, but the size of the frame agrees with MessageSetSize, > > > because there are only 2022 bytes left in the frame. The bytes after > the > > > crc field to the end of the frame don't checksum to the value of the > crc > > > field either. > > > > > > (topics) 00 00 00 01 > > > TopicName 00 1F (+ 31 character topic name) > > > (partitions) 00 00 00 01 > > > Partition 00 00 00 02 > > > ErrorCode 00 00 > > > HighwaterMarkOffset 00 00 00 00 00 3F 5B 17 > > > MessageSetSize 00 00 08 00 > > > Offset 00 00 00 00 00 3F 5B 16 > > > MessageSize 00 00 10 B3 > > > Crc B0 13 F4 E3 > > > MagicByte 00 > > > Attributes 02 > > > Key FF FF FF FF > > > Value 00 00 10 A5 (+ 2022 bytes left in frame) > > > > > > What's going on here, what have I misunderstood? > > > > > > yours, > > > Theo > > > > > > On Sat, Feb 28, 2015 at 8:35 PM, Theo Hultberg <t...@iconara.net> > wrote: > > > > > >> Hi, > > >> > > >> Mostly out of curiosity I'm writing a Kafka client, and I'm getting > > stuck > > >> at decoding fetch responses. Most of the time everything goes fine, > but > > >> quite often I get frames back that seem to be wrong. I'm sure I've > > >> misunderstood something about the spec, so maybe someone can get me on > > the > > >> right track. > > >> > > >> I get frames where the MessageSetSize is 2048, but it contains > messages > > >> whose MessageSize is larger than this. The MessageSetSize is always > > 2048, > > >> which makes me suspicious. > > >> > > >> I've tried feeding these message sets to the Java client (into > > >> MemoryRecords.readableRecords), but when I iterate over it, it is > empty. > > >> Further inspecting what it does with the bytes I see that it gets an > > >> EOFException and stops iterating, and I don't really understand what's > > >> going on. > > >> > > >> I'm doing these fetch requests against a cluster running 0.8.2.1. > > >> > > >> yours, > > >> Theo > > >> > > > > > > > > >
