ignoreUnknownValues is supported for BATCH_LOADS as well. On Fri, May 7, 2021 at 7:08 AM Matthew Ouyang <[email protected]> wrote:
> Thank you for responding Evan. It looks like these options will only work > for STREAMING_INSERTS. Are there any options for BATCH_LOADS, and if not > are there any plans for it? > > On Thu, May 6, 2021 at 6:11 PM Evan Galpin <[email protected]> wrote: > >> Hey Matthew, >> >> I believe you might also need to use the “ignoreUnknownValues”[1] or >> skipInvalidRows[2] options depending on your use case if your goal is to >> allow valid entities to succeed even if invalid entities exist and >> separately process failed entities via “getFailedResults”. You could also >> consider introducing additional validation before the Write step if the >> failure modes are predictable. >> >> Thanks, >> Evan >> >> [1] >> >> https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#ignoreUnknownValues-- >> [2] >> >> https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#skipInvalidRows-- >> >> On Thu, May 6, 2021 at 18:01 Matthew Ouyang <[email protected]> >> wrote: >> >>> I am loading a batch load of records with BigQueryIO.Write, but because >>> some records don't match the target table schema the entire and the write >>> step fails and nothing gets written to the table. Is there a way for >>> records that do match the target table schema to be inserted, and the >>> records that don't match don't cause the entire step to fail? I noticed >>> BigQueryIO.Write returns a WriteResult that has a method getFailedInserts. >>> Will that meet my needs? >>> >>
