Ok Jhon. But If i want to implement an alternative for myself. What do you recommend in order to get the message and send it to other target (you said is possible)? taking in mind that we re using the kafka connector which is a java transformation which is invoke for python
El mié, 6 dic 2023 a las 11:23, John Casey (<[email protected]>) escribió: > For the moment, yes. > > On Wed, Dec 6, 2023 at 11:21 AM Juan Romero <[email protected]> wrote: > >> Thanks John. Is it the same case if i want to write in a postgres table >> with the sql connector? >> >> El mié, 6 dic 2023 a las 11:05, John Casey (<[email protected]>) >> escribió: >> >>> It is, but it's not possible to to take an existing transform, and >>> simply configure it to do this. >>> >>> For example (and this is what I'm doing), it's possible to write a >>> transform that tries to write to kafka, and upon failure, emits the failure >>> to an alternate pcollection. >>> >>> It's not possible (yet) to take an existing PTransform that's part of >>> the library, and configure it to do something other than simply retrying >>> failures >>> >>> On Wed, Dec 6, 2023 at 10:44 AM Juan Romero <[email protected]> wrote: >>> >>>> But , is it not possible to get the message that can't reach the target >>>> sink and put it in another target (eg: kafka error topic where we can >>>> verify which messages failed to be delivered to the target)? >>>> >>>> >>>> El mié, 6 dic 2023 a las 10:40, John Casey via user (< >>>> [email protected]>) escribió: >>>> >>>>> I'm currently implementing improvements on Kafka, File, Spanner, and >>>>> Bigtable IOs. >>>>> >>>>> I'm planning on tackling PubSub and BQ next year. >>>>> >>>>> All of this is still in progress though, so there aren't easy >>>>> workarounds for the moment. >>>>> >>>>> On Tue, Dec 5, 2023 at 5:56 PM Robert Bradshaw <[email protected]> >>>>> wrote: >>>>> >>>>>> Currently error handling is implemented on sinks in an ad-hoc basis >>>>>> (if at all) but John (cc'd) is looking at improving things here. >>>>>> >>>>>> On Mon, Dec 4, 2023 at 10:25 AM Juan Romero <[email protected]> >>>>>> wrote: >>>>>> > >>>>>> > Hi guys. I want to ask you about how to deal with the scenario when >>>>>> the target sink (eg: jdbc, kafka, bigquery, pubsub etc) fails for any >>>>>> reason and i don't want to lost the message and create a bottleneck with >>>>>> many errors due an hypothetical target sink problem, and i want to use >>>>>> with_excpetion_handling in order to get the message that failing to reach >>>>>> the target and send the message to an other error topic. Any idea to >>>>>> solve >>>>>> this scenario? >>>>>> >>>>>
