Kevin Peterson created BEAM-1826:
------------------------------------
Summary: Allow BigqueryIO to forward errors
Key: BEAM-1826
URL: https://issues.apache.org/jira/browse/BEAM-1826
Project: Beam
Issue Type: Bug
Components: sdk-java-gcp
Reporter: Kevin Peterson
Assignee: Daniel Halperin
Priority: Minor
Most sinks are terminal - data ends at the sink. While on occasion the sink may
temporarily fail due to resource unavailability, it will eventually succeed.
However, some have strict requirements on this input format. In these cases,
retries will never succeed, and continuous retrying will eventually lead to
pipeline failure.
The primary use case I have in mind is streaming data to a sink such as
BigQuery, where data of the wrong format could fail on insert.
It would be useful to be able to set a side output or downstream transform from
Bigquery which can receive failed rows where retry will never fix the issue,
and allow them to be persisted to a different output which is more permissive
of the output, to prevent data loss.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
