carlpayne commented on issue #23291: URL: https://github.com/apache/beam/issues/23291#issuecomment-1330485358
@reuvenlax re: `The failure comes when we first open the connection to BigQuery`, from my experiments using the code I posted earlier, this might not always be the case, e.g. if we try to write a TableRow to BigQuery which is missing a required field, it _does_ get caught by the error handling [here](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java#L437), which means in theory we do have access to the `AppendRowsContext` and the failed `ProtoRows` at this point. Would it be possible to extend the error handling here to not only check for `failedContext.getError() instanceof Exceptions.AppendSerializtionError` but also instances of `InvalidArgumentException`? This would allow us to write the failed rows back out to `getFailedStorageApiInserts` without the 1k retries -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
