Tord Sætren created BEAM-9917:
---------------------------------
Summary: BigQueryBatchFileLoads dynamic destination
Key: BEAM-9917
URL: https://issues.apache.org/jira/browse/BEAM-9917
Project: Beam
Issue Type: Bug
Components: sdk-py-core
Affects Versions: 2.20.0
Reporter: Tord Sætren
I am trying to use BigQueryBatchFileLoads to upload data from pubsub. It works
fine for a single table, but when I try to use a dynamic destination such as
destination=lambda elem: "my_project:my_dataset." + elem["sensor_key"],
it just makes a new table for each time the triggering_frequency procs. I know
it makes temporary tables before loading it all into one, but it never loads
them. It just creates more and more tables.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
