BostjanBozic commented on issue #26196:
URL: https://github.com/apache/beam/issues/26196#issuecomment-2017498974
@liferoad thanks for the help here. I was first trying to fetch info using
just plain `ReadFromBigQuery`, but it seems that `ReadFromBigQueryRequest`
together with `ReadAllFromBigQuery` is required. This seems to be fetching data
(does not work locally though, only when running on Dataflow, which is also
stated in code), but I guess the output is a bit different then if we would
just use `ReadFromBigQuery`, so I will have to troubleshoot this part. But at
least read works for now :)
```
lookup_table = (
pipeline
| "Retrigger BigQuery Data Read" >> PeriodicImpulse(fire_interval=600)
| "Prepare BigQuery Data Read"
>> beam.Map(
lambda x: ReadFromBigQueryRequest(
query="SELECT col1, col2, col3 FROM test.test_table;",
use_standard_sql=True,
flatten_results=False,
)
)
| "Reading BigQuery Data"
>> ReadAllFromBigQuery(
temp_dataset="temp"
)
| "Reshuffling BigQuery Data" >> beam.Reshuffle()
| "Keying BigQuery Data" >> beam.Map(lambda row: (row.get("col1"), row))
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
