No, I think David answered the specific question that I asked i.e. is it
okay (or not) for operators other than sinks and side outputs to do I/O.
Purging DLQ entries is something we'll need to be able to do anyway (for
some scenarios - aside from successful checkpoint retries) and I
specifically wa
I am not 100% certain that David is talking about the same pattern of usage
that you are Tom.
David, the pattern Tom is talking about is something like this...
try {
do something with record
} catch (SomeException e) {
push record to DLQ
}
My concern is that if we have a different failure,
Thank you David.
In the case we have in mind it should only happen literally on the very
rare Exception i.e. in some cases if somehow an uncaught exception occurs,
we want to send the record to a DLQ and handle the retry manually Vs
checkpointing and restarting.
Regards,
Tom.
On Sun, Jul 26, 2
Every job is required to have a sink, but there's no requirement that all
output be done via sinks. It's not uncommon, and doesn't have to cause
problems, to have other operators that do I/O.
What can be problematic, however, is doing blocking I/O. While your user
function is blocked, the function
Hi.
What are the negative side effects of (for example) a filter function
occasionally making a call out to a DB ? Is this a big no-no and should all
outputs be done through sinks and side outputs, no exceptions ?
Regards,
Tom.
