jrmccluskey commented on PR #39236: URL: https://github.com/apache/beam/pull/39236#issuecomment-6148133419
> I'm a little iffy about this approach. It requires the input PCollection schema to be the full union of all destination table schemas. And the input schema is still fixed, so can't meet the need of new tables containing new columns, or existing table schema updates. > > Also tables with identical column names will need to agree on their types and modes (e.g. table1 has colA as a nullable INT32 but table2 has colA as an array of Strings) > > I think a more robust long term solution would need a semi-structured type like Variant so rows can include only what they need (see #38251) Yeah I haven't been thrilled with the options across the cross-language boundary, the core issue is that the row coder between the Python SDK and the Java SDK is fixed at pipeline construction time. Not being able to handle dynamic schemas kinda undercuts the whole point of dynamic destinations. I agree that the Variant type probably solves this for us, the only question is if we want to provide a "dynamic destinations, static schemas" version of this before we implement Variant (might unblock some use cases but will likely not be streaming update compatible since we'll be changing coders around.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
