Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in low volume suppression buffers

2019-10-22 Thread Richard Yu
Hi John, Reading the KIP over, I actually think that this will be an excellent extension to what we have now. I've read the discussion for that KIP and it looks like Jonathan never mentioned how time would be tracked (wall-clock based that is) while this KIP I believe proposes a feasible approach

Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in low volume suppression buffers

2019-10-22 Thread John Roesler
Hi Richard, Thanks for the KIP. After reading it over, I'm wondering if you saw this one: https://cwiki.apache.org/confluence/display/KAFKA/KIP-424%3A+Allow+suppression+of+intermediate+events+based+on+wall+clock+time It seems like it overlaps with your use case, where you're willing to get

Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in low volume suppression buffers

2019-10-20 Thread Richard Yu
Hi Bill, John, I've made some changes to the KIP after some thinking. I've come up with a reasonable solution, I believe, to relieve the problem that is associated with low traffic suppression buffers. Would be great if I can get your input on this. :) Cheers, Richard On Fri, Oct 18, 2019 at

Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in low volume suppression buffers

2019-10-18 Thread Richard Yu
Hi Bill, Thanks for the input! TBH, I am think that suppression buffers are not used *in response *to low traffic conditions. Rather, we are trying to fix the situation when low traffic conditions occur in a suppression buffer (for example, previously, the same suppression buffer had a decent

Re: [DISCUSS] KIP-539: Implement mechanism to flush out records in low volume suppression buffers

2019-10-18 Thread Bill Bejeck
Hi Richard, Thanks for the KIP proposal. I understand the situation you are describing. But in my mind, if there is a low traffic condition and you need to keep records going downstream at regular intervals, I'm wondering if using suppression is the correct approach. IMHO it seems it would be

[DISCUSS] KIP-539: Implement mechanism to flush out records in low volume suppression buffers

2019-10-17 Thread Richard Yu
Hi all, I wish to discuss this KIP which would help us in resolving some issues we have with suppression buffers. Below is the link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-539%3A+Implement+mechanism+to+flush+out+records+in+low+volume+suppression+buffers @John Roesler if you have