rizenfrmtheashes commented on issue #23670: URL: https://github.com/apache/beam/issues/23670#issuecomment-1281393215
It's entirely possible that during an autoscale event, work items may get replayed, and the third party side-effects that involve the copy job and the remove jobs may get replayed. specifically for work items that trigger a remove job before being reset since it didn't reach the end of the pipeline and considered "committed". Regardless, there should be a way to handle/track this, or make sure that the load to temp table -> copy to destination -> remove temp table happens as a single commit phase, and allow duplication. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
