[ 
https://issues.apache.org/jira/browse/KAFKA-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-6352:
-----------------------------------
    Description: 
We have observed some users beginning to use the new message format before 
their clients have been upgraded. As we know, this can cause a lot of memory 
pressure due to the fact that we down-convert the full response in memory. 
Currently we do this down-conversion prior to enqueuing the response to be sent 
by the Processor, which may cause more heap utilization than needed with a 
steady queue size. 

A possible improvement is to delay the down-conversion until the response is 
actually ready to be sent. Even better would be to do the down-conversion in a 
streaming fashion, converting only as much as is needed at any time. One 
potential drawback is that this moves the down-conversion into the network 
threads.

  was:
We have observed some users beginning to use the new message format before 
their clients have been upgraded. As we know, this can cause a lot of memory 
pressure due to the fact that we lose zero-copy in order to down-convert the 
messages. Currently we do down-conversion prior to enqueuing the response to be 
sent by the Processor. This may lead to more heap utilization than needed with 
a steady queue size. 

A possible improvement is to delay the down-conversion until the response is 
actually ready to be sent. Even better would be to do the down-conversion in a 
streaming fashion, converting only as much as is needed at any time. One 
potential drawback is that this moves the down-conversion into the network 
threads.


> Delay message down-conversion until response is ready to be sent
> ----------------------------------------------------------------
>
>                 Key: KAFKA-6352
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6352
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Jason Gustafson
>
> We have observed some users beginning to use the new message format before 
> their clients have been upgraded. As we know, this can cause a lot of memory 
> pressure due to the fact that we down-convert the full response in memory. 
> Currently we do this down-conversion prior to enqueuing the response to be 
> sent by the Processor, which may cause more heap utilization than needed with 
> a steady queue size. 
> A possible improvement is to delay the down-conversion until the response is 
> actually ready to be sent. Even better would be to do the down-conversion in 
> a streaming fashion, converting only as much as is needed at any time. One 
> potential drawback is that this moves the down-conversion into the network 
> threads.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to