mxm commented on issue #13406: URL: https://github.com/apache/iceberg/issues/13406#issuecomment-3089391804
Hey @peach12345! Thank you for using Iceberg and the Dynamic Sink. This is a tricky subject. Using `UnknownType` won't work because Iceberg schema evolution doesn't let you convert to a specific type later on. The reason is that converting `UnknownType` would make the type narrower, and we generally only allow widening types (e.g. `int => long`). I suppose we could implement support for creating `UnknownType` fields and update their type later, if the type changed only once from unknown to known. In case old data were to arrive, a null value would simply be written for the column. What you are doing right now is (the "workaround") is also not bad. With the Dynamic Sink you can decide to skip writing the column entirely, if its type is not known and there is no data for it. Once you know the type based on the data, you create it. Again, old records which don't have the data yet can simply write null values for the field. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
