Re: [VOTE] SEP-20: Samza on Kubernetes

2019-11-07 Thread Jagadish Venkatraman
+1 binding. Thanks Weiqing for driving this! On Thursday, November 7, 2019, Xinyu Liu wrote: > +1 (binding). > > Thanks, > Xinyu > > On Thu, Nov 7, 2019 at 10:50 AM Weiqing Yang > wrote: > > > Hi All, > > > > The feedback from the discussion thread: > >

Re: Occasional checkpoint mismatch on Samza task reload

2019-11-07 Thread Malcolm McFarland
Also, is there a way to produce this error, ie if we added extra messages to the __checkpoint topics? Malcolm McFarland Cavulus This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any unauthorized or improper disclosure, copying, distribution, or use of the contents of this message

Re: Samza compaction policy

2019-11-07 Thread Malcolm McFarland
Actually, do you have an example of some appropriate settings for Kafka to ensure that compaction is behaving correctly for the Samza checkpoint topics? Cheers, Malcolm McFarland Cavulus This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any unauthorized or improper disclosure,

Re: Samza compaction policy

2019-11-07 Thread Malcolm McFarland
Hi Prateek, As far as I can gather, we are indeed seeing duplicate keys in the checkpoint topics. We definitely have cleanup.policy=compact on all of these topics. When you suggest that I check the topic partition size graph, is this just the checkpoint topics, or any that have

Re: Occasional checkpoint mismatch on Samza task reload

2019-11-07 Thread Malcolm McFarland
Hi Bharath, Interesting to hear about the TTL. Is this a Kafka topic-level configuration? This really does seem to only occur when the stream processors are restarted, and only occasionally at that. We've been looking at increasing the yarn.nodemanager.sleep-delay-before-sigkill.ms and

Re: [VOTE] SEP-20: Samza on Kubernetes

2019-11-07 Thread Xinyu Liu
+1 (binding). Thanks, Xinyu On Thu, Nov 7, 2019 at 10:50 AM Weiqing Yang wrote: > Hi All, > > The feedback from the discussion thread: > http://mail-archives.apache.org/mod_mbox/samza-dev/201911.mbox/browser is > positive. This is a call for a vote for SEP-20: Samza on Kubernetes < > >