Thanks. But it's still unclear how one could construct the query that reads from external data sources supported by BigQuery. Sayak Paul | sayak.dev
On Tue, Jan 11, 2022 at 7:00 AM Chamikara Jayalath <[email protected]> wrote: > For a general Python BQ example see here: > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py > (and specify the 'query' parameter of 'ReadFromBigQuery' transform instead > of the 'table' parameter. > > I haven't tried with a BigTable external table as I mentioned. > > Thanks, > Cham > > On Mon, Jan 10, 2022 at 5:13 PM Sayak Paul <[email protected]> wrote: > >> Thanks! Could you provide a reference? >> Sayak Paul | sayak.dev >> >> >> >> On Tue, Jan 11, 2022 at 12:52 AM Chamikara Jayalath <[email protected]> >> wrote: >> >>> I haven't tried it, but you can try including such a table definition in >>> a BQ query. It should work if BQ supports exporting results of such a query >>> to GCS using export jobs. >>> >>> Thanks, >>> Cham >>> >>> On Mon, Jan 10, 2022 at 5:02 AM Sayak Paul <[email protected]> >>> wrote: >>> >>>> Hi folks, >>>> >>>> BigQuery supports reading [1] from external sources such as BigTable >>>> [2]. I was wondering if it's possible to use it to read from BigQuery in a >>>> Beam pipeline. There's no explicit argument called >>>> `external_table_definition` in `ReadFromBigQuery`. >>>> >>>> *References* >>>> >>>> [1] https://cloud.google.com/bigquery/external-table-definition >>>> [2] https://cloud.google.com/bigquery/external-data-bigtable >>>> >>>> Sayak Paul | sayak.dev >>>> >>>>
