[ https://issues.apache.org/jira/browse/KAFKA-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Gustafson updated KAFKA-6352: ----------------------------------- Description: We have observed some users beginning to use the new message format before their clients have been upgraded. As we know, this can cause a lot of memory pressure due to the fact that we down-convert the full response in memory. Currently we do this down-conversion prior to enqueuing the response to be sent by the Processor, which may cause more heap utilization than needed with a steady queue size. A possible improvement is to delay the down-conversion until the response is actually ready to be sent. Even better would be to do the down-conversion in a streaming fashion, converting only as much as is needed at any time. One potential drawback is that this moves the down-conversion into the network threads. was: We have observed some users beginning to use the new message format before their clients have been upgraded. As we know, this can cause a lot of memory pressure due to the fact that we lose zero-copy in order to down-convert the messages. Currently we do down-conversion prior to enqueuing the response to be sent by the Processor. This may lead to more heap utilization than needed with a steady queue size. A possible improvement is to delay the down-conversion until the response is actually ready to be sent. Even better would be to do the down-conversion in a streaming fashion, converting only as much as is needed at any time. One potential drawback is that this moves the down-conversion into the network threads. > Delay message down-conversion until response is ready to be sent > ---------------------------------------------------------------- > > Key: KAFKA-6352 > URL: https://issues.apache.org/jira/browse/KAFKA-6352 > Project: Kafka > Issue Type: Improvement > Reporter: Jason Gustafson > > We have observed some users beginning to use the new message format before > their clients have been upgraded. As we know, this can cause a lot of memory > pressure due to the fact that we down-convert the full response in memory. > Currently we do this down-conversion prior to enqueuing the response to be > sent by the Processor, which may cause more heap utilization than needed with > a steady queue size. > A possible improvement is to delay the down-conversion until the response is > actually ready to be sent. Even better would be to do the down-conversion in > a streaming fashion, converting only as much as is needed at any time. One > potential drawback is that this moves the down-conversion into the network > threads. -- This message was sent by Atlassian JIRA (v6.4.14#64029)