max.message.size is not enforced for compressed messages
--------------------------------------------------------
Key: KAFKA-275
URL: https://issues.apache.org/jira/browse/KAFKA-275
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.7
Reporter: Neha Narkhede
The max.message.size check is not performed for compressed messages, but only
for each message that forms a compressed message. Due to this, even if the
max.message.size is set to 1MB, the producer can technically send n 1MB
messages as one compressed message. This can cause memory issues on the server
as well as deserialization issues on the consumer. The consumer's fetch size
has to be > max.message.size in order to be able to read data. If one message
is larger than the fetch.size, the consumer will throw an exception and cannot
proceed until the fetch.size is increased.
Due to this bug, even if the fetch.size > max.message.size, the consumer can
still get stuck on a message that is larger than max.message.size.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
