szlta commented on PR #4662: URL: https://github.com/apache/iceberg/pull/4662#issuecomment-1115035033
Hey @szehon-ho this was just intended to be a quick fix so we don't throw exceptions for the case you mentioned too. Currently if we want to collapse the matching partition columns we will hit the following issue: - suppose we'd have 1000: data, 1001: data, 1002: id in the combined partition type - Partitioning.partitionType could collapse this into 1001: data, 1002: id (leaving out the last seen "data" as going back in specs) - this will be the "read schema" when reading the manifest files (as the underlying avro files) - for spec0, id matching will think that although the file schema has 1000: data, but the read schema doesn't, this reading will just return null for 1001: data and null for 1002: id leaving us no info the collapse upon.. Perhaps if we could rework the re-addition of "data" column to spec so that no new field ID is generated for it? I'm not sure if this will cause other issues though.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
