kkdoon commented on issue #28554: URL: https://github.com/apache/beam/issues/28554#issuecomment-1747311798
ok, adding a flag for drain behavior sounds reasonable. Lastly, I just wanted to double check with what issues do you see if we instead skip the watermark hold check inside `flushData` and do it inside `notifyCheckpointComplete` (only when drain is initiated) ? That way we still maintain the contract of flushing after checkpoint is completed. And in this case if final checkpoint fails, its still ok since drain will fail and pipeline will continue running (and user can retry again later). And if checkpoint is successful but stable DoFn fails, then drain will finish successfully with a warning message (to restore last savepoint and re-run the pipeline since some elements were not processed). If watermark skip approach doesn't seem correct, then i will update my [PR](https://github.com/apache/beam/pull/28567) to implement approach A. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
