Hi there, I will take a look first thing i get home. Zahari
> On 31 Oct 2018, at 18:23, Mayuresh Gharat <gharatmayures...@gmail.com> wrote: > > Hi Colin, Zahari, > > Wanted to check if you can review the patch and let me know, if we need to > make any changes? > > Thanks, > > Mayuresh > > On Fri, Oct 26, 2018 at 1:41 PM Zahari Dichev <zaharidic...@gmail.com> > wrote: > >> Thanks for participating the discussion. Indeed, I learned quite a lot. >> Will take a look at the patch as well and spend some time hunting for some >> other interesting issue to work on :) >> >> Cheers, >> Zahari >> >>> On Fri, Oct 26, 2018 at 8:49 PM Colin McCabe <cmcc...@apache.org> wrote: >>> >>> Hi Zahari, >>> >>> I think we can retire the KIP, since the KAFKA-7548 patch should solve >> the >>> issue without any changes that require a KIP. This is actually the best >>> thing we could do for our users, since things will "just work" more >>> efficiently without a lot of configuration knobs. >>> >>> I think you did an excellent job raising this issue and discussing it. >>> It's a very good contribution to the project even if you don't end up >>> writing the patch yourself. I'm going to take a look at the patch today. >>> If you want to take a look, that would also be good. >>> >>> best, >>> Colin >>> >>> >>>> On Thu, Oct 25, 2018, at 12:25, Zahari Dichev wrote: >>>> Hi there Mayuresh, >>>> >>>> Great to heat that this is actually working well in production for some >>>> time now. I have changed the details of the KIP to reflect the fact >> that >>> as >>>> already discussed - we do not really need any kind of configuration as >>> this >>>> data should not be thrown away at all. Submitting a PR sounds great, >>>> although I feel a bit jealous you (LinkedIn) beat me to my first kafka >>>> commit ;) Not sure how things stand with the voting process ? >>>> >>>> Zahari >>>> >>>> >>>> >>>> On Thu, Oct 25, 2018 at 7:39 PM Mayuresh Gharat < >>> gharatmayures...@gmail.com> >>>> wrote: >>>> >>>>> Hi Colin/Zahari, >>>>> >>>>> I have created a ticket for the similar/same feature : >>>>> https://issues.apache.org/jira/browse/KAFKA-7548 >>>>> We (Linkedin) had a use case in Samza at Linkedin when they moved >> from >>> the >>>>> SimpleConsumer to KafkaConsumer and they wanted to do this pause and >>> resume >>>>> pattern. >>>>> They realized there was performance degradation when they started >> using >>>>> KafkaConsumer.assign() and pausing and unPausing partitions. We >>> realized >>>>> that not throwing away the prefetched data for paused partitions >> might >>>>> improve the performance. We wrote a benchmark (I can share it if >>> needed) to >>>>> prove this. I have attached the findings in the ticket. >>>>> We have been running the hotfix internally for quite a while now. >> When >>>>> samza ran this fix in production, they realized 30% improvement in >>> there >>>>> app performance. >>>>> I have the patch ready on our internal branch and would like to >> submit >>> a PR >>>>> for this on the above ticket asap. >>>>> I am not sure, if we need a separate config for this as we haven't >>> seen a >>>>> lot of memory overhead due to this in our systems. We have had this >>> running >>>>> in production for a considerable amount of time without any issues. >>>>> It would be great if you guys can review the PR once its up and see >> if >>> that >>>>> satisfies your requirement. If it doesn't then we can think more on >> the >>>>> config driven approach. >>>>> Thoughts?? >>>>> >>>>> Thanks, >>>>> >>>>> Mayuresh >>>>> >>>>> >>>>> On Thu, Oct 25, 2018 at 8:21 AM Colin McCabe <cmcc...@apache.org> >>> wrote: >>>>> >>>>>> Hi Zahari, >>>>>> >>>>>> One question we didn't figure out earlier was who would actually >> want >>>>> this >>>>>> cached data to be thrown away. If there's nobody who actually >> wants >>>>> this, >>>>>> then perhaps we can simplify the proposal by just unconditionally >>>>> retaining >>>>>> the cache until the partition is resumed, or we unsubscribe from >> the >>>>>> partition. This would avoid adding a new configuration. >>>>>> >>>>>> best, >>>>>> Colin >>>>>> >>>>>> >>>>>>> On Sun, Oct 21, 2018, at 11:54, Zahari Dichev wrote: >>>>>>> Hi there, although it has been discussed briefly already in this >>> thread >>>>>>> < >>>>>> >>>>> >>> >> https://lists.apache.org/thread.html/fbb7e9ccc41084fc2ff8612e6edf307fb400f806126b644d383b4a64@%3Cdev.kafka.apache.org%3E >>>>>>> , >>>>>>> I decided to follow the process and initiate a DISCUSS thread. >>> Comments >>>>>>> and >>>>>>> suggestions are more than welcome. >>>>>>> >>>>>>> >>>>>>> Zahari Dichev >>>>>> >>>>> >>>>> >>>>> -- >>>>> -Regards, >>>>> Mayuresh R. Gharat >>>>> (862) 250-7125 >>>>> >>> >> > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125
