Hello The BigQuery Storage Write API is GA since October 2021 (docs at [1]). BEAM-11648 was created to adapt the Beam BigQuery sink.
That Jira issue is still marked as open. However, it seems that the functionality has already been added. The Javadoc has the STORAGE_WRITE_API method added since release 2.29.0 (see [2]). The latest release (2.35.0), removed the notion that it is an experimental API (at BigQuery side) and also added STORAGE_API_AT_LEAST_ONCE method [3]. However the Beam documentation at [4] does not mention the Storage Write API option at all. Can we consider this development done and start using these features? What would be the best approach for a Python pipeline? I found BEAM-10917 for the Storage Read API with Python SDK. That Jira is also open, but again seems functionality has been added. As from release 2.34.0 I see that it is possible to provide method = DIRECT_READ to use the Storage Read API [5] (always in Arvo it seems, not clear how you could use Arrow). But didn’t find anything for the Storage Write API. Is it better (and even possible) to use the Java BigQuery sink using the multi language features? Apologies for these annoying questions. As a dataflow user, I am a bit lost to understand what is the reference (Google docs are limited and refer to Beam, but seems the docs lag a bit versus the code). Thanks a lot! Bruno [1] https://cloud.google.com/bigquery/docs/write-api <https://cloud.google.com/bigquery/docs/write-api> [2] https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.Method.html <https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.Method.html> [3] https://beam.apache.org/releases/javadoc/2.35.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.Method.html <https://beam.apache.org/releases/javadoc/2.35.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.Method.html> [4] https://beam.apache.org/documentation/io/built-in/google-bigquery/#writing-to-bigquery <https://beam.apache.org/documentation/io/built-in/google-bigquery/#writing-to-bigquery> [5] https://beam.apache.org/releases/pydoc/2.34.0/apache_beam.io.gcp.bigquery.html <https://beam.apache.org/releases/pydoc/2.34.0/apache_beam.io.gcp.bigquery.html>
