Thanks for the KIP, Dong. In the current threading model, compression is done by the user threads, therefore the producer sender thread can focus on IO. With the proposed changes, does that mean the producer sender thread will have to do all the compression as well? Would this become a performance bottleneck?
Thanks, Jiangjie (Becket) Qin On Thu, Apr 12, 2018 at 1:41 AM, Dong Lin <lindon...@gmail.com> wrote: > Hey Ted, > > Thanks for your comments. With the proposed solution in the KIP, the memory > is only allocated once for the given message, which is the same as the > existing implementation. The serialized message will be moved from > per-topic queue to per-partition queue without incurring additional memory > overhead. Does this address your question? > > Thanks, > Dong > > > On Wed, Apr 11, 2018 at 9:36 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > Looks like per-topic queue is introduced. > > > > In terms of memory consumption, how does the KIP allocate memory > > between per-topic > > queue and per-partition queue ? > > > > Thanks > > > > On Wed, Apr 11, 2018 at 8:50 PM, Dong Lin <lindon...@gmail.com> wrote: > > > > > Hi all, > > > > > > I have created KIP-286: producer.send() should not block on metadata > > > update. See > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 286%3A+producer.send%28%29+should+not+block+on+metadata+update > > > . > > > > > > The KIP intends to improve user-experience of producer.send() when > > metadata > > > is not available. It is related but different from the previous > > discussion > > > in KAFKA-3539 in the sense that user still has the option of letting > > > producer.send() block on full producer queue. > > > > > > Comments are welcome! > > > > > > Thanks, > > > Dong > > > > > >
