xuanswe opened a new issue, #34000: URL: https://github.com/apache/beam/issues/34000
### What would you like to happen? According to https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub#dead-letter-topics > Dataflow might acknowledge messages before the pipeline fully processes the data. Specifically, Dataflow acknowledges messages after they are successfully processed by the first fused stage and side effects of that processing have been written to persistent storage. If the pipeline has multiple fused stages and failures occur at any point after the first stage, the messages are already acknowledged and don't go to the dead-letter topic. First of all, I am new to Dataflow and PubSubIO, so if I am wrong on something, please correct me. I understand that, if there is an uncaught exception (ex. due to a bug) after the first fused stage, the message is already acked and will never be redelivered after fixing the bug. If I am wrong, please explain to me where I am wrong and please explain the way dataflow prevents data loss in this situation. If it's true, could you please tell me the solution to ensure **100%** that I do not lose any data for the mentioned situation or anything? If there is currently no solution yet. What do you think if PubSubIO supports a method `withAutoAckOnSuccess(false)` like below? ``` PubsubIO.readMessagesWithAttributes() .fromSubscription("projects/projectId/subscriptions/subscriptionId") .withAutoAckOnSuccess(false) ``` Then developer will manually ack the message at the last stage of the pipeline manually. ### Issue Priority Priority: 2 (default / most feature requests should be filed as P2) ### Issue Components - [ ] Component: Python SDK - [ ] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [x] Component: IO connector - [ ] Component: Beam YAML - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Infrastructure - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Samza Runner - [ ] Component: Twister2 Runner - [ ] Component: Hazelcast Jet Runner - [x] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
