Re: [DISCUSS] KIP-283: Efficient Memory Usage for Down-Conversion

Hi all, I have updated the KIP to reflect changes from the discussion in this thread. In particular, the KIP now recommends adding a configuration that allows disabling any kind of message down-conversion for client `FetchRequest`, instead of having to specify the minimum compatibility. I also

Re: [DISCUSS] KIP-283: Efficient Memory Usage for Down-Conversion

Hi Jun, Yes, that is true. Ideally, we should be able to down-convert only the first message batch in the request handling thread and delay everything else till the network thread. I have not thought through all the details of how we could do this but at first glance this seems tricky to

Re: [DISCUSS] KIP-283: Efficient Memory Usage for Down-Conversion

Hi, Dhruvil, Thanks for the KIP. Looks good me to overall. Just one comment below. "To prevent this from happening, we will not delay down-conversion of the first partition in the response. We will down-convert all messages of the first partition in the I/O thread (like we do today), and only

Re: [DISCUSS] KIP-283: Efficient Memory Usage for Down-Conversion

Thanks Dhruvil. That makes sense. On Thu, Apr 12, 2018 at 2:49 AM, Dhruvil Shah wrote: > Hi Rajini, > > Thanks for the comments. > > Which config parameters are these referring to? > > This refers to a proposal that was later rejected. I have removed this goal > from the

Re: [DISCUSS] KIP-283: Efficient Memory Usage for Down-Conversion

Hi Jason, 1. The motivation for adding the configuration was that even though we are improving memory usage, there is still overhead involved with down-conversion - the broker still has to do extra work which some users might want to eliminate completely. The proposal also moves the

Re: [DISCUSS] KIP-283: Efficient Memory Usage for Down-Conversion

Hi Rajini, Thanks for the comments. Which config parameters are these referring to? This refers to a proposal that was later rejected. I have removed this goal from the KIP as it is no longer valid. What exactly is a chunk going to be I have updated the KIP to remove references to the fixed

Re: [DISCUSS] KIP-283: Efficient Memory Usage for Down-Conversion

Hey Dhruvil, Thanks for the KIP. Looks good overall. I have a few questions about the new configs: 1. I'm mainly wondering how necessary the configs are given the improvements in this KIP to reduce memory pressure from down-conversion. The reason I ask is that we'll be stuck with this config for

Re: [DISCUSS] KIP-283: Efficient Memory Usage for Down-Conversion

Hi Dhruvil, Thanks for the KIP. This is a great improvement to reduce OOMs in brokers during down-conversion. Just a couple of minor questions: The goals state: "*Provide appropriate configuration parameters to manage maximum memory usage during down-conversion on the broker.*" Which config

Re: [DISCUSS] KIP-283: Efficient Memory Usage for Down-Conversion

Hi Ted, Thanks for the comments. *>> bq. we can perform down-conversion when Records.writeTo is called.>> Wouldn't this delay the network thread (though maybe the duration is short)>> ?* Yes, this is noted in the Cons section. I think we have a precedent for this in the `SSLTransportLayer`

Re: [DISCUSS] KIP-283: Efficient Memory Usage for Down-Conversion

bq. we can perform down-conversion when Records.writeTo is called. Wouldn't this delay the network thread (though maybe the duration is short) ? Can you expand on the structure of LazyDownConvertedRecords in more detail ? bq. even if it exceeds fetch.max.bytes I did a brief search but didn't

Re: [DISCUSS] KIP-283: Efficient Memory Usage for Down-Conversion

I fixed the diagrams - let me know if you are still having trouble seeing them. Thanks, Dhruvil On Fri, Apr 6, 2018 at 3:05 PM, Ted Yu wrote: > The two embedded diagrams seem broken. > > Can you double check ? > > Thanks > > On Fri, Apr 6, 2018 at 2:56 PM, Dhruvil Shah

Re: [DISCUSS] KIP-283: Efficient Memory Usage for Down-Conversion

The two embedded diagrams seem broken. Can you double check ? Thanks On Fri, Apr 6, 2018 at 2:56 PM, Dhruvil Shah wrote: > Hi, > > I created a KIP to help mitigate out of memory issues during > down-conversion. The KIP proposes introducing a configuration that can >