[
https://issues.apache.org/jira/browse/KAFKA-5062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16052451#comment-16052451
]
Colin P. McCabe commented on KAFKA-5062:
----------------------------------------
I agree that there are some intriguing questions here about whether the 100 MB
{{socket.request.max.bytes}} is being honored in every case. However, even if
we assume that this limit is honored in every case, 100 MB is a big buffer to
allocate because someone sent a nonsense request. The JVM will not handle such
large memory allocations well, especially on smaller heaps.
I think we can be smarter about just reading the very first part of the Kafka
request with the API key and API version fields. That should rule out most of
the bad requests which we will get.
I'm posting this up for jenkins to see if any unit tests break. It might need
to be tweaked for SSL, etc.
> Kafka brokers can accept malformed requests which allocate gigabytes of memory
> ------------------------------------------------------------------------------
>
> Key: KAFKA-5062
> URL: https://issues.apache.org/jira/browse/KAFKA-5062
> Project: Kafka
> Issue Type: Bug
> Reporter: Apurva Mehta
> Assignee: Colin P. McCabe
>
> In some circumstances, it is possible to cause a Kafka broker to allocate
> massive amounts of memory by writing malformed bytes to the brokers port.
> In investigating an issue, we saw byte arrays on the kafka heap upto 1.8
> gigabytes, the first 360 bytes of which were non kafka requests -- an
> application was writing the wrong data to kafka, causing the broker to
> interpret the request size as 1.8GB and then allocate that amount. Apart from
> the first 360 bytes, the rest of the 1.8GB byte array was null.
> We have a socket.request.max.bytes set at 100MB to protect against this kind
> of thing, but somehow that limit is not always respected. We need to
> investigate why and fix it.
> cc [~rnpridgeon], [~ijuma], [~gwenshap], [~cmccabe]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
