Re: Occasional checkpoint mismatch on Samza task reload

2019-11-07 Thread Malcolm McFarland
Also, is there a way to produce this error, ie if we added extra messages to the __checkpoint topics? Malcolm McFarland Cavulus This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any unauthorized or improper disclosure, copying, distribution, or use of the contents of this message

Re: Occasional checkpoint mismatch on Samza task reload

2019-11-07 Thread Malcolm McFarland
Hi Bharath, Interesting to hear about the TTL. Is this a Kafka topic-level configuration? This really does seem to only occur when the stream processors are restarted, and only occasionally at that. We've been looking at increasing the yarn.nodemanager.sleep-delay-before-sigkill.ms and

Re: Occasional checkpoint mismatch on Samza task reload

2019-10-31 Thread Bharath Kumara Subramanian
Hi Malcolm, The warning is not particularly related to your restarts. It happens when the offset requested by the consumer no longer exists on the broker. It can typically happen if you are consuming from a topic with TTL enabled and the time retention kicked in and broker purged older offsets.

Occasional checkpoint mismatch on Samza task reload

2019-10-31 Thread Malcolm McFarland
Hey folks, We're running Samza 0.14.1 on Hadoop 2.7.6. Every once in a while while restarting an application, the process will come up with some variation on this error: INFO Validating offset for topic and partition [,] WARN While refreshing brokers for [,]: