>> Was kafka designed for a specific message size range? Kafka consumer reads a message from the socket into memory. If a message is large enough to cause OutOfMemoryException, then the Kafka consumer is unable to return more messages from the socket byte buffer. This can be fixed by enabling the Kafka consumers to have a 'streaming' API, where such large messages could be read in a piecemeal fashion. But it is tricky and we don't have that feature yet.
To avoid your Kafka consumer from getting into a bad state due to a large message, you can set "max.message.size" to the largest possible message size on your producer. Any message larger than that never enters the Kafka cluster and hence never reaches a Kafka consumer. >> > Seeing as it is used to aggregate log messages, is it safe to say message > sizes of 2-100K are reasonable and won't cause any issues? 100K message sizes should work fine. Thanks, Neha On Wed, Dec 21, 2011 at 12:15 PM, S Ahmed <sahmed1...@gmail.com> wrote: > Was kafka designed for a specific message size range? > > Seeing as it is used to aggregate log messages, is it safe to say message > sizes of 2-100K are reasonable and won't cause any issues?
