slachiewicz opened a new pull request, #912: URL: https://github.com/apache/incubator-xtable/pull/912
Part of #911. ## What is the purpose of the pull request Delta assigns column mapping ids to struct fields only, so a map key, a map value and a list element have no id of their own. When `delta.enableIcebergCompatV2` is set, Delta does assign them, writes them into the parquet file, and records them on the enclosing field under `delta.columnMapping.nested.ids`, keyed by the path the child takes in the file, such as `col-1234.key`. This reads that metadata and puts the ids on the converted fields. It matters because Delta 3.x writes parquet field ids where Delta 2.4 wrote none, and Iceberg applies a name mapping only to files that carry no ids at all. Once any id is present, the nested children of a collection have to resolve by id too, and today they have none. #911 has the footers from both Delta versions and the failure that follows. ## Brief change log - `DeltaSchemaExtractor` threads a field's `delta.columnMapping.nested.ids` metadata through the conversion and assigns the ids it holds to the map key, map value and list element fields. - `IcebergSchemaExtractor` needed no change: it already prefers a field's own id over one it generates. ## Verify this pull request This change added tests and can be verified as follows: - `TestDeltaSchemaExtractor#testNestedFieldIdsInDeltaSchema` covers a map and a list carrying the metadata, and a field without it, which keeps today's behaviour of leaving the children unassigned. - `./mvnw -pl xtable-core test -Dtest=TestDeltaSchemaExtractor,TestIcebergSchemaExtractor` runs 21 tests, all green. The end to end effect is only observable on Delta 3.x, since Delta 2.4 has no IcebergCompatV2 and writes no parquet field ids, so on this branch the change is inert. #908, which moves the build to Spark 3.5 and Delta 3.3, is where `ITConversionController#testColumnMappingEnabledDeltaToIceberg` exercises it. *This change was created with AI assistance.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
