Here's an example of a frame that looks malformed, I've split the bytes apart and annotated the pieces with the names from the Kafka protocol guide ( https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse ).
Notice that the MessageSetSize is 0x00000800 (2048), but the MessageSize is 0x000010B3 (4275). The size prefix of the Value field agrees with MessageSize, but the size of the frame agrees with MessageSetSize, because there are only 2022 bytes left in the frame. The bytes after the crc field to the end of the frame don't checksum to the value of the crc field either. (topics) 00 00 00 01 TopicName 00 1F (+ 31 character topic name) (partitions) 00 00 00 01 Partition 00 00 00 02 ErrorCode 00 00 HighwaterMarkOffset 00 00 00 00 00 3F 5B 17 MessageSetSize 00 00 08 00 Offset 00 00 00 00 00 3F 5B 16 MessageSize 00 00 10 B3 Crc B0 13 F4 E3 MagicByte 00 Attributes 02 Key FF FF FF FF Value 00 00 10 A5 (+ 2022 bytes left in frame) What's going on here, what have I misunderstood? yours, Theo On Sat, Feb 28, 2015 at 8:35 PM, Theo Hultberg <t...@iconara.net> wrote: > Hi, > > Mostly out of curiosity I'm writing a Kafka client, and I'm getting stuck > at decoding fetch responses. Most of the time everything goes fine, but > quite often I get frames back that seem to be wrong. I'm sure I've > misunderstood something about the spec, so maybe someone can get me on the > right track. > > I get frames where the MessageSetSize is 2048, but it contains messages > whose MessageSize is larger than this. The MessageSetSize is always 2048, > which makes me suspicious. > > I've tried feeding these message sets to the Java client (into > MemoryRecords.readableRecords), but when I iterate over it, it is empty. > Further inspecting what it does with the bytes I see that it gets an > EOFException and stops iterating, and I don't really understand what's > going on. > > I'm doing these fetch requests against a cluster running 0.8.2.1. > > yours, > Theo >
