Hi all,
I have updated the KIP to reflect changes from the discussion in this
thread. In particular, the KIP now recommends adding a configuration that
allows disabling any kind of message down-conversion for client
`FetchRequest`, instead of having to specify the minimum compatibility.
I also ad
Hi Jun,
Yes, that is true. Ideally, we should be able to down-convert only the
first message batch in the request handling thread and delay everything
else till the network thread. I have not thought through all the details of
how we could do this but at first glance this seems tricky to implement
Hi, Dhruvil,
Thanks for the KIP. Looks good me to overall. Just one comment below.
"To prevent this from happening, we will not delay down-conversion of the
first partition in the response. We will down-convert all messages of the
first partition in the I/O thread (like we do today), and only del
Thanks Dhruvil. That makes sense.
On Thu, Apr 12, 2018 at 2:49 AM, Dhruvil Shah wrote:
> Hi Rajini,
>
> Thanks for the comments.
>
> Which config parameters are these referring to?
>
> This refers to a proposal that was later rejected. I have removed this goal
> from the KIP as it is no longer v
Hi Jason,
1. The motivation for adding the configuration was that even though we are
improving memory usage, there is still overhead involved with
down-conversion - the broker still has to do extra work which some users
might want to eliminate completely. The proposal also moves the
down-conversio
Hi Rajini,
Thanks for the comments.
Which config parameters are these referring to?
This refers to a proposal that was later rejected. I have removed this goal
from the KIP as it is no longer valid.
What exactly is a chunk going to be
I have updated the KIP to remove references to the fixed bu
Hey Dhruvil,
Thanks for the KIP. Looks good overall. I have a few questions about the
new configs:
1. I'm mainly wondering how necessary the configs are given the
improvements in this KIP to reduce memory pressure from down-conversion.
The reason I ask is that we'll be stuck with this config for
Hi Dhruvil,
Thanks for the KIP. This is a great improvement to reduce OOMs in brokers
during down-conversion.
Just a couple of minor questions:
The goals state: "*Provide appropriate configuration parameters to manage
maximum memory usage during down-conversion on the broker.*"
Which config para
Hi Ted,
Thanks for the comments.
*>> bq. we can perform down-conversion when Records.writeTo is called.>>
Wouldn't this delay the network thread (though maybe the duration is
short)>> ?*
Yes, this is noted in the Cons section. I think we have a precedent for
this in the `SSLTransportLayer` impl
bq. we can perform down-conversion when Records.writeTo is called.
Wouldn't this delay the network thread (though maybe the duration is short)
?
Can you expand on the structure of LazyDownConvertedRecords in more detail ?
bq. even if it exceeds fetch.max.bytes
I did a brief search but didn't se
I fixed the diagrams - let me know if you are still having trouble seeing
them.
Thanks,
Dhruvil
On Fri, Apr 6, 2018 at 3:05 PM, Ted Yu wrote:
> The two embedded diagrams seem broken.
>
> Can you double check ?
>
> Thanks
>
> On Fri, Apr 6, 2018 at 2:56 PM, Dhruvil Shah wrote:
>
> > Hi,
> >
> >
The two embedded diagrams seem broken.
Can you double check ?
Thanks
On Fri, Apr 6, 2018 at 2:56 PM, Dhruvil Shah wrote:
> Hi,
>
> I created a KIP to help mitigate out of memory issues during
> down-conversion. The KIP proposes introducing a configuration that can
> prevent down-conversions al
Hi,
I created a KIP to help mitigate out of memory issues during
down-conversion. The KIP proposes introducing a configuration that can
prevent down-conversions altogether, and also describes a design for
efficient memory usage for down-conversion.
https://cwiki.apache.org/confluence/display/KAFK
13 matches
Mail list logo