VardhanThigle commented on issue #34160: URL: https://github.com/apache/beam/issues/34160#issuecomment-2702843885
> [@VardhanThigle](https://github.com/VardhanThigle) So do we just add the same retry logic that is present in the `JdbcIO` or do we add make like an enum in `CassandraIO` with say `HandlingAction` [`RETRY`,`SKIP`, `FAIL`] and then tweak the `ReadFn` to handle some retrying logic? There are various ways to go about this 1. [easy but less ideal] We could just log and re-throw and let the runner (like Dataflow) retry. I would appreciate if we also produce metrics here. 2. We could let the user plug-in an exception handler for reader (writer side, the user has more control as the user wirtes the save or delete functions) I would lean away from enums as different exceptions would need different handling. Also, side request, there are many things that could be made pluggable in Beam's CassandraIO THe connection manager - currently it's very tied to 3.0 interfaces. and 4.0 driver comes with a rich set of retry level configuration at driver level. There's a meta question here, if and when we shift to 4.0 intefaces in CassandraIO for beam, would we still need an exception handling - IMHO yes, atleast we should rethrow, the current approach of silent logging makes data migrationpipeline complete with a success state even if I turn down Cassandra for example where the pipeline should fail with a few (possible configurable) retries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
