GitHub user Susmit07 edited a discussion: Pekko Connector Fault Tolerance Processing
Hi Team, I tested this functionality - Prepare 2000 files in a test location local inside the pod - Deploy the app on Kubernetes with multiple pods using the Cluster Singleton setup. Ensure that only one pod is processing files at a time. (which we achieved) - While the first pod is processing files, kill the pod to simulate failure. - Monitor the logs of the next pod that takes over. It should resume processing from the last processed file. But actually its processing from the first one again, in other words no offset is maintained. Does pekko out of the box supports exactly once or we need to maintain the state in a PersistentVolumes / DB like a key value store along with Pekko Persistence module Awaiting response GitHub link: https://github.com/apache/pekko-connectors/discussions/834 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
