Hey Matthew, I believe you might also need to use the “ignoreUnknownValues”[1] or skipInvalidRows[2] options depending on your use case if your goal is to allow valid entities to succeed even if invalid entities exist and separately process failed entities via “getFailedResults”. You could also consider introducing additional validation before the Write step if the failure modes are predictable.
Thanks, Evan [1] https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#ignoreUnknownValues-- [2] https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#skipInvalidRows-- On Thu, May 6, 2021 at 18:01 Matthew Ouyang <[email protected]> wrote: > I am loading a batch load of records with BigQueryIO.Write, but because > some records don't match the target table schema the entire and the write > step fails and nothing gets written to the table. Is there a way for > records that do match the target table schema to be inserted, and the > records that don't match don't cause the entire step to fail? I noticed > BigQueryIO.Write returns a WriteResult that has a method getFailedInserts. > Will that meet my needs? >
