[
https://issues.apache.org/jira/browse/BEAM-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Kirpichov closed BEAM-1834.
----------------------------------
Resolution: Fixed
Fix Version/s: 2.2.0
Support for data-dependent schemas has been added a pretty long time ago via
BigQueryIO.write().to(DynamicDestinations).
> Bigquery Write validation doesn't work well with ValueInSingleWindow
> --------------------------------------------------------------------
>
> Key: BEAM-1834
> URL: https://issues.apache.org/jira/browse/BEAM-1834
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-gcp
> Reporter: Kevin Peterson
> Fix For: 2.2.0
>
>
> I am using the new {{Write<T> to(SerializableFunction<ValueInSingleWindow<T>,
> String> tableSpecFunction)}} function to write data to different Bigquery
> tables depending on the values. I'm my case, the values can have a different
> schema (it starts as an {{Any}} encoded protobuf, which I parse and expand to
> a {{TableRow}} object).
> Since the tables have different schemas, the existing implementation of
> {{withSchema}} doesn't work.
> Some options:
> # Allow {{CreateDisposition.CREATE_NEVER}} in this situation. Failed inserts
> from a missing table just fail (and eventually pass through via BEAM-190).
> # Add a new {{withSchema(SerializableFunction<ValueInSingleWindow<T>,
> TableSchema>}} function.
> I think eventually both of the above should be allowable configurations, but
> just one will unblock my current error. Happy to implement, given some
> guidance on design preferences.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
