Kevin Peterson created BEAM-1834:
------------------------------------
Summary: Bigquery Write validation doesn't work well with
ValueInSingleWindow
Key: BEAM-1834
URL: https://issues.apache.org/jira/browse/BEAM-1834
Project: Beam
Issue Type: Bug
Components: sdk-java-gcp
Reporter: Kevin Peterson
Assignee: Daniel Halperin
I am using the new {{Write<T> to(SerializableFunction<ValueInSingleWindow<T>,
String> tableSpecFunction)}} function to write data to different Bigquery
tables depending on the values. I'm my case, the values can have a different
schema (it starts as an {{Any}} encoded protobuf, which I parse and expand to a
{{TableRow}} object).
Since the tables have different schemas, the existing implementation of
{{withSchema}} doesn't work.
Some options:
# Allow {{CreateDisposition.CREATE_NEVER}} in this situation. Failed inserts
from a missing table just fail (and eventually pass through via BEAM-190).
# Add a new {{withSchema(SerializableFunction<ValueInSingleWindow<T>,
TableSchema>}} function.
I think eventually both of the above should be allowable configurations, but
just one will unblock my current error. Happy to implement, given some guidance
on design preferences.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
