boyuanzz commented on pull request #13795: URL: https://github.com/apache/beam/pull/13795#issuecomment-774351114
> Won't a failure in finalization callback mean that messages are not ACKed so they will be replayed? To my knowledge, it depends on the source implementation details. Taking KafkaIO as an example, the `finalizeCheckpoint` does commit the offset but this commit has no effect on current reading(since the reader will track current position), Within such implementation, the replayed happens when you stop your pipeline then restart it, and you want to read from last committed offset. I want to discuss more about error handling on bundle finalization. The SDK can choose to fail the bundle directly or SDK can choose to return a response with error to runner, then let the runner to decide whether to fail the bundle. Currently in python sdk, the SDK will return the response with error instead of failing the bundle. But the Java SDK fails the bundle directly. I'm leaning to let the runner make the decision when exception happens(ignore/retry/fail the execution). What do you think? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
