ahmedabu98 opened a new issue, #27478:
URL: https://github.com/apache/beam/issues/27478
### What would you like to happen?
Auto-schema updates allow Storage API writes to adapt to BigQuery schema
changes (e.g. required field relaxes to nullable). However, currently it
doesn't support schema field additions (ie. a new column being added). Best to
explain with an example:
Say we are writing to a table with schema 1: {field1, field2}
and while we are writing, the table updates to schema 2: {field1, field2,
field3}
If we start writing TableRows that match schema 2, they will fail and go to
the PCollection's DLQ. This fails not at the data insertion step, but when we
convert the TableRows to proto prior to that using
[StorageApiDynamicDestinationsTableRow](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiDynamicDestinationsTableRow.java).
This dynamic destinations object contains a `MessageConverter` that is
instantiated **once** for each destination. The destination table's schema is
fetched in the constructor of this MessageConverter object and is never updated
later:
https://github.com/apache/beam/blob/843e7fdb239e0a61b47e39ce635c37221cd09f4e/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiDynamicDestinationsTableRow.java#L93-L123
There should be a way to update the schema when appropriate.
### Issue Priority
Priority: 3 (nice-to-have improvement)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
