Pablo thanks so much I will explore this method then for STREAMING_INSERTS <https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.Method.html#STREAMING_INSERTS>, this answer worth much for us :)
thanks. On Tue, May 7, 2019 at 1:05 PM Pablo Estrada <[email protected]> wrote: > Hi Andres! > You can definitely do streaming inserts using the Java SDK. This is > available with BigQueryIO.write(). Specifically, you can use the > `withMethod`[1] call to specify whether you want batch loads or streaming > inserts. If you specify streaming inserts, Beam should insert rows as they > come in bundles. > Hope that helps > -P. > > [1] > https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#withMethod-org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.Method- > > On Tue, May 7, 2019 at 9:58 AM Andres Angel < > [email protected]> wrote: > >> Hello everyone, >> >> I need to use BigQuery inserts within my beam pipeline, hence I know well >> the built-in IO options offer `BigQueryIO`, however this will insert in a >> batch fashion to BQ creating underneath a BQ load job. I instead need to >> trigger a streaming insert into BQ, and I was reviewing the Java SDK >> documentation but seems like this is not possible. >> >> In the other hand, I have the python SDK and I found this GitHub >> documentation >> <https://beam.apache.org/releases/pydoc/2.4.0/_modules/apache_beam/io/gcp/bigquery.html> >> code where they are using a method *InsertAll >> <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py>* >> which >> is apparently what I need. If this is official I would like to know if >> there is a naive fashion to trigger stream inserts in BQ using the Java SDK. >> >> thanks so much for your feedback >> AU >> >
