ahmedabu98 commented on PR #39236: URL: https://github.com/apache/beam/pull/39236#issuecomment-6134009021
I'm a little iffy about this approach. It requires the input PCollection schema to be the full union of all destination table schemas. And the input schema is still fixed, so can't meet the need of new tables containing new columns, or existing table schema updates. Also tables with identical column names will need to agree on their types and modes (e.g. table1 has colA as a nullable INT32 but table2 has colA as an array of Strings) I think a more robust long term solution would need a semi-structured type like Variant so rows can include only what they need (see https://github.com/apache/beam/issues/38251) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
