rdblue commented on issue #40: Add external schema mappings for files written with name-based schemas URL: https://github.com/apache/incubator-iceberg/issues/40#issuecomment-467232457 @rdsr, sorry for the delay. I was thinking about a solution more like the alternative you proposed, but I was thinking that this would work using just Avro schemas, so no need to convert from Iceberg to Avro. Iceberg already has field IDs, the question is how to match those up with the Avro schema in a data file. We also don't want to change the schema from the file too much because it is required to correctly read the data. So converting to Iceberg, then back to Avro is much more risky than transforming Avro to Avro+ids. I like your idea to have a some mapping callback, similar to `NextID` but that can be implemented by something that contains the mapping. Hopefully, that API could be used for Avro and Parquet ID mapping since we'd like to be able to do the same for parquet-avro files, eventually.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
