Hi Jun, Yes, that is true. Ideally, we should be able to down-convert only the first message batch in the request handling thread and delay everything else till the network thread. I have not thought through all the details of how we could do this but at first glance this seems tricky to implement, given that `FetchResponse.PartitionData` for the first partition will be a combination of `MemoryRecords` and `LazyDownConvertedRecords`. I will think about this a bit more to see if I can come up with a clean abstraction, and will also add it to the KIP.
Thanks, Dhruvil On Mon, Apr 16, 2018 at 6:07 PM, Jun Rao <j...@confluent.io> wrote: > Hi, Dhruvil, > > Thanks for the KIP. Looks good me to overall. Just one comment below. > > "To prevent this from happening, we will not delay down-conversion of the > first partition in the response. We will down-convert all messages of the > first partition in the I/O thread (like we do today), and only delay > down-conversion for subsequent partitions." It seems that we can further > optimize this by only down-converting the first message set in the first > partition in the request handling threads? > > Jun > > > On Fri, Apr 6, 2018 at 2:56 PM, Dhruvil Shah <dhru...@confluent.io> wrote: > > > Hi, > > > > I created a KIP to help mitigate out of memory issues during > > down-conversion. The KIP proposes introducing a configuration that can > > prevent down-conversions altogether, and also describes a design for > > efficient memory usage for down-conversion. > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 283%3A+Efficient+Memory+Usage+for+Down-Conversion > > > > Suggestions and feedback are welcome! > > > > Thanks, > > Dhruvil > > >
