Hi I'd like to know how SimpleConsumer.fetch and ByteBufferMessageSet.iterartor work in terms of dealing with GZIP'd messages
is it right to say that SimpleConsumer.fetch will fetch upto the amount specified in the fetch size regardless whether a complete GZIP batch has been retrieved, and ByteBufferMessageSet.iterartor is responsible for converting the fetched MessageSet into a set of complete MessageAndOffset objects and disgarding any incomplete batch due to fetch size limit? I'm asking because I'm regularly getting InvalidMessageSizeException after a consumer fetch, and I can see this topic being discussed in other mail threads already, and I've also used the DumpLogSegments to check the kafka file reported in the exception, nothing negative reported by the tool in terms of the offset I'm using GZIP compression to send msgs to kafka, and a limit of 3MB of raw data batch has been imposed and the batch size will be smaller once GZIP'ed to kafka On the consumer the fetch size is set to 10 MB to allow for plenty of leeway to fetch at least 1 full batch of GZIP'd content what else can be wrong in this setup ? -- Rgds Ray
