slachiewicz opened a new issue, #911:
URL: https://github.com/apache/incubator-xtable/issues/911
### Search before asking
I searched the open and closed issues; nothing covers this.
### Describe the problem
Converting a Delta table that has column mapping enabled and contains a
`map` or `array` column produces an Iceberg table that cannot be read, once the
Delta writer is 3.x. Reading it fails in Iceberg's Parquet reader:
```
java.lang.IllegalArgumentException: [col-127c1280-..., key_value, key]
required binary key (STRING) is not in the store: [...]
```
`ITConversionController#testColumnMappingEnabledDeltaToIceberg` covers this
path and passes today, because the build is on Delta 2.4. It fails against
Delta 3.3, which is what surfaced this.
### Root cause
Delta 3.x writes parquet field ids; Delta 2.4 did not. The difference is
visible in the footer of a file written by the same code against each version,
for a table with `delta.columnMapping.mode = name`:
```
delta-core 2.4.0 optional group col-0284b2db-... (MAP) { repeated group
key_value { ... } }
delta-spark 3.3.3 optional group col-d660aa27-... (MAP) = 2 { repeated
group key_value { ... } }
```
XTable's Delta to Iceberg path relies on Iceberg's name mapping, which
`IcebergConversionTarget#createAndSetNameMapping` populates with Delta's
physical names. Iceberg applies a name mapping only to files that carry no
field ids at all. Delta 3.x's ids therefore switch Iceberg to id based
resolution, and Delta assigns column mapping ids to struct fields only, never
to a map key, a map value or a list element. Those nested fields end up with no
id in the file and no name mapping to fall back on, so they cannot be resolved.
A map key is required, so the read fails rather than returning nulls.
Delta's own answer to this is `delta.enableIcebergCompatV2`, which hooks
`DeltaParquetWriteSupport` specifically to, in its words, "write nested field
IDs for list and map types to the parquet schema". With that enabled the ids
are in the file, and Delta records them per field under
`delta.columnMapping.nested.ids`, keyed by the path the child takes in the
file, for example `col-1234.key`.
### What this is not
Ruled out by experiment:
- Iceberg 1.10.2 in place of 1.9.2.
- Parquet 1.13.1 in place of 1.15.2.
- An id or name bug on the XTable side. The Delta schema json and the id to
physical name map that `IcebergSchemaExtractor` builds are identical between
Delta 2.4 and 3.3.
### Suggested fix
Two parts, of which this issue covers the first:
1. `DeltaSchemaExtractor` should read `delta.columnMapping.nested.ids` and
put those ids on the converted map key, map value and list element fields.
`IcebergSchemaExtractor` already prefers a field's own id over one it
generates, so the Iceberg schema then matches the file.
2. Decide what to do for a source table that does not enable
IcebergCompatV2. On Delta 3.x such a table is not convertible to a readable
Iceberg table when it contains collections, which is worth documenting whatever
the outcome.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
*This issue was created with AI assistance.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
