It is, but it's not possible to to take an existing transform, and simply configure it to do this.
For example (and this is what I'm doing), it's possible to write a transform that tries to write to kafka, and upon failure, emits the failure to an alternate pcollection. It's not possible (yet) to take an existing PTransform that's part of the library, and configure it to do something other than simply retrying failures On Wed, Dec 6, 2023 at 10:44 AM Juan Romero <[email protected]> wrote: > But , is it not possible to get the message that can't reach the target > sink and put it in another target (eg: kafka error topic where we can > verify which messages failed to be delivered to the target)? > > > El mié, 6 dic 2023 a las 10:40, John Casey via user (<[email protected]>) > escribió: > >> I'm currently implementing improvements on Kafka, File, Spanner, and >> Bigtable IOs. >> >> I'm planning on tackling PubSub and BQ next year. >> >> All of this is still in progress though, so there aren't easy workarounds >> for the moment. >> >> On Tue, Dec 5, 2023 at 5:56 PM Robert Bradshaw <[email protected]> >> wrote: >> >>> Currently error handling is implemented on sinks in an ad-hoc basis >>> (if at all) but John (cc'd) is looking at improving things here. >>> >>> On Mon, Dec 4, 2023 at 10:25 AM Juan Romero <[email protected]> wrote: >>> > >>> > Hi guys. I want to ask you about how to deal with the scenario when >>> the target sink (eg: jdbc, kafka, bigquery, pubsub etc) fails for any >>> reason and i don't want to lost the message and create a bottleneck with >>> many errors due an hypothetical target sink problem, and i want to use >>> with_excpetion_handling in order to get the message that failing to reach >>> the target and send the message to an other error topic. Any idea to solve >>> this scenario? >>> >>
