Thank you for responding Evan. It looks like these options will only work for STREAMING_INSERTS. Are there any options for BATCH_LOADS, and if not are there any plans for it?
On Thu, May 6, 2021 at 6:11 PM Evan Galpin <[email protected]> wrote: > Hey Matthew, > > I believe you might also need to use the “ignoreUnknownValues”[1] or > skipInvalidRows[2] options depending on your use case if your goal is to > allow valid entities to succeed even if invalid entities exist and > separately process failed entities via “getFailedResults”. You could also > consider introducing additional validation before the Write step if the > failure modes are predictable. > > Thanks, > Evan > > [1] > > https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#ignoreUnknownValues-- > [2] > > https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#skipInvalidRows-- > > On Thu, May 6, 2021 at 18:01 Matthew Ouyang <[email protected]> > wrote: > >> I am loading a batch load of records with BigQueryIO.Write, but because >> some records don't match the target table schema the entire and the write >> step fails and nothing gets written to the table. Is there a way for >> records that do match the target table schema to be inserted, and the >> records that don't match don't cause the entire step to fail? I noticed >> BigQueryIO.Write returns a WriteResult that has a method getFailedInserts. >> Will that meet my needs? >> >
