That the message set size was always the same when this happened made me start looking for that number, and it turned out that was what I had set as the MaxBytes field of the requests. This make me think that what happens is that the next message to fetch is larger than this, but it's sent anyway. Is this correct? Is this how it is supposed to work? I'm looking through other clients to see how they deal with this, and I think I might have found the same thing in the Go client ( https://github.com/Shopify/sarama/blob/master/message_set.go#L72-L76), and the corresponding comment in the protocol guide. Is what I have found the thing described as "As an optimization the server is allowed to return a partial message at the end of the message set. Clients should handle this case."?
yours, Theo On Sat, Feb 28, 2015 at 10:19 PM, Theo Hultberg <t...@iconara.net> wrote: > Here's an example of a frame that looks malformed, I've split the bytes > apart and annotated the pieces with the names from the Kafka protocol guide > ( > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse > ). > > Notice that the MessageSetSize is 0x00000800 (2048), but the MessageSize > is 0x000010B3 (4275). The size prefix of the Value field agrees with > MessageSize, but the size of the frame agrees with MessageSetSize, > because there are only 2022 bytes left in the frame. The bytes after the > crc field to the end of the frame don't checksum to the value of the crc > field either. > > (topics) 00 00 00 01 > TopicName 00 1F (+ 31 character topic name) > (partitions) 00 00 00 01 > Partition 00 00 00 02 > ErrorCode 00 00 > HighwaterMarkOffset 00 00 00 00 00 3F 5B 17 > MessageSetSize 00 00 08 00 > Offset 00 00 00 00 00 3F 5B 16 > MessageSize 00 00 10 B3 > Crc B0 13 F4 E3 > MagicByte 00 > Attributes 02 > Key FF FF FF FF > Value 00 00 10 A5 (+ 2022 bytes left in frame) > > What's going on here, what have I misunderstood? > > yours, > Theo > > On Sat, Feb 28, 2015 at 8:35 PM, Theo Hultberg <t...@iconara.net> wrote: > >> Hi, >> >> Mostly out of curiosity I'm writing a Kafka client, and I'm getting stuck >> at decoding fetch responses. Most of the time everything goes fine, but >> quite often I get frames back that seem to be wrong. I'm sure I've >> misunderstood something about the spec, so maybe someone can get me on the >> right track. >> >> I get frames where the MessageSetSize is 2048, but it contains messages >> whose MessageSize is larger than this. The MessageSetSize is always 2048, >> which makes me suspicious. >> >> I've tried feeding these message sets to the Java client (into >> MemoryRecords.readableRecords), but when I iterate over it, it is empty. >> Further inspecting what it does with the bytes I see that it gets an >> EOFException and stops iterating, and I don't really understand what's >> going on. >> >> I'm doing these fetch requests against a cluster running 0.8.2.1. >> >> yours, >> Theo >> > >
