stankiewicz commented on issue #34465: URL: https://github.com/apache/beam/issues/34465#issuecomment-2765910219
some notes, Underlying KafkaProducer usually have some defaults on retry but eventually it will throw kafka exception KafkaIO's KafkaWriter is capturing outcome of writing via Callback. On finish bundle, if there were exceptions, it will raise IO exception which will retry cause runner to potentially retry the bundle. Dataflow Streaming retries forever to guarantee no data loss as there is no custom error handling in IO for network connection. There is Bad Record handling via tag. Dataflow Batch will cancel job after 4 failed tries. Potential implementation: similarly to baderrorhandler router, we could allow kafkaerrorhandler router. such handler should give full message so the user can build DLQ and retry mechanism on their own if needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
