Unfortunately, there's no way to leverage the existing cross language
connector in python.
Your options are somewhat limited, in my opinion.
Option 1 (My Recommendation) : Implement a DoFn that checks your data
quality before sending it to KafkaIO. If it fails the quality check, send
it to some
Ok Jhon. But If i want to implement an alternative for myself. What do you
recommend in order to get the message and send it to other target (you said
is possible)? taking in mind that we re using the kafka connector which is
a java transformation which is invoke for python
El mié, 6 dic 2023 a
For the moment, yes.
On Wed, Dec 6, 2023 at 11:21 AM Juan Romero wrote:
> Thanks John. Is it the same case if i want to write in a postgres table
> with the sql connector?
>
> El mié, 6 dic 2023 a las 11:05, John Casey ()
> escribió:
>
>> It is, but it's not possible to to take an existing
Thanks John. Is it the same case if i want to write in a postgres table
with the sql connector?
El mié, 6 dic 2023 a las 11:05, John Casey ()
escribió:
> It is, but it's not possible to to take an existing transform, and simply
> configure it to do this.
>
> For example (and this is what I'm
It is, but it's not possible to to take an existing transform, and simply
configure it to do this.
For example (and this is what I'm doing), it's possible to write a
transform that tries to write to kafka, and upon failure, emits the failure
to an alternate pcollection.
It's not possible (yet)
But , is it not possible to get the message that can't reach the target
sink and put it in another target (eg: kafka error topic where we can
verify which messages failed to be delivered to the target)?
El mié, 6 dic 2023 a las 10:40, John Casey via user ()
escribió:
> I'm currently
I'm currently implementing improvements on Kafka, File, Spanner, and
Bigtable IOs.
I'm planning on tackling PubSub and BQ next year.
All of this is still in progress though, so there aren't easy workarounds
for the moment.
On Tue, Dec 5, 2023 at 5:56 PM Robert Bradshaw wrote:
> Currently
Currently error handling is implemented on sinks in an ad-hoc basis
(if at all) but John (cc'd) is looking at improving things here.
On Mon, Dec 4, 2023 at 10:25 AM Juan Romero wrote:
>
> Hi guys. I want to ask you about how to deal with the scenario when the
> target sink (eg: jdbc, kafka,
Hi guys. I want to ask you about how to deal with the scenario when the
target sink (eg: jdbc, kafka, bigquery, pubsub etc) fails for any reason
and i don't want to lost the message and create a bottleneck with many
errors due an hypothetical target sink problem, and i want to use
