salimTutuncu opened a new issue, #977: URL: https://github.com/apache/incubator-xtable/issues/977
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues. ### Please describe the bug 🐞 ### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues. ### Please describe the bug 🐞 When `delta.columnMapping.mode=name` is enabled on an existing Delta table and a column is then renamed, the Iceberg target produced by the Delta Kernel conversion source (`useDeltaKernel=true`) cannot resolve the renamed column in the files written before column mapping was enabled. Delta reads all three values. From the Iceberg target, Spark 4.1.1 / Iceberg 1.11.0 and DuckDB 1.5.6 return NULL for the two old rows, and PyIceberg 0.12.0 fails: ```text ValueError: Could not find field with name: name ``` ```text expected (Delta source): (1, USA, John Smith) (2, USA, Mary Johnson) (6, Italy, Luca Verdi) actual (Spark, DuckDB): (1, USA, NULL) (2, USA, NULL) (6, Italy, Luca Verdi) actual (PyIceberg): read fails on the old physical column `name` ``` Reproduced on unmodified `main` 9d9f9b3 (`0.5.0-SNAPSHOT`) in both `FULL` and `INCREMENTAL` mode, and on `0.4.0-incubating`. The renamed column keeps Iceberg field ID 3 across the incremental sync and the table UUID is unchanged, so this is not the field-ID change of #711; the generated name mapping simply lacks the old physical name. The sync reports `SUCCESS`. ### Why this is not a duplicate of #711 or #765 #711 reports that an incremental sync turns a Delta rename into drop + add (field ID 2 → 5 there). I see that on `0.3.0-incubating` (3 → 4, all names NULL), but on `0.4.0-incubating` and `main` the ID is preserved (3 → 3) and the failure is a missing name-mapping alias, a different mechanism. #765 / #766 fixed column-mapped Delta → Iceberg conversion by adding `InternalField.storageName` and writing the physical names into the Iceberg name mapping. That fix is in the Standalone `DeltaSchemaExtractor` and is not present in the Kernel source path. #911 covers nested field IDs under column mapping and notes that Iceberg applies a name mapping only to files that carry no field IDs, which is exactly the pre-mapping files in this fixture. #936 is the target-side counterpart (Kernel target and `physicalName`). This is the "fix that only exists on the Standalone path" case that #886 prerequisite 2 asks to verify before Kernel becomes the default. ### Root cause `DeltaKernelSchemaExtractor.toInternalSchema` reads `delta.columnMapping.physicalName` into a local `storageName` (L155-158) and uses it only to build `nestedIdPath` (L173); the `InternalField.builder()` call (L174-181) sets `name`, `fieldId`, `parentPath`, `schema`, `defaultValue` and never `storageName`: https://github.com/apache/incubator-xtable/blob/9d9f9b3ced22adea03001b4b5e547b51fa7c8adc/xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelSchemaExtractor.java#L155-L181 The Standalone `DeltaSchemaExtractor` calls `.storageName(storageName)` at the same position (L191), added by #766: https://github.com/apache/incubator-xtable/blob/9d9f9b3ced22adea03001b4b5e547b51fa7c8adc/xtable-core/src/main/java/org/apache/xtable/delta/DeltaSchemaExtractor.java#L188-L196 `IcebergSchemaExtractor` records an id → storage-name entry only when `getStorageName()` is non-null, and `IcebergConversionTarget` updates the Iceberg name mapping from that map. With the Kernel source the map stays empty, so the mapping for the renamed column is `{"field-id":3,"names":["new_name"]}`, while the two old Parquet files carry no field IDs and the physical column `name` (Delta keeps the existing name as physical name when mapping is enabled late). The file written after the rename carries field IDs and resolves by ID, which is why only the old rows are NULL. Diagnostic, not a repair: appending `name` to field 3's names in an in-memory copy of the metadata makes PyIceberg return all three rows; no stored metadata was modified. Control: with column mapping enabled at table creation, all three names resolve on `main` and `0.4.0` (the files carry Parquet field IDs), but every partition value is NULL, which is #A. Complete schema history and decoded name mapping of the `main` incremental target: ```json {"current-schema-id":2, "schemas":[{"type":"struct","schema-id":0,"fields":[]}, {"type":"struct","schema-id":1,"fields":[{"id":1,"name":"id","type":"int"},{"id":2,"name":"country","type":"string"},{"id":3,"name":"name","type":"string"}]}, {"type":"struct","schema-id":2,"fields":[{"id":1,"name":"id","type":"int"},{"id":2,"name":"country","type":"string"},{"id":3,"name":"new_name","type":"string"}]}], "schema.name-mapping.default":[{"field-id":1,"names":["id"]},{"field-id":2,"names":["country"]},{"field-id":3,"names":["new_name"]}]} ``` ### To Reproduce Environment: as in #A (sync host Spark 3.5.5 / JDK 17, `main` 9d9f9b3 built with Java 11, Delta Kernel 4.0.0, delta-storage 3.3.0, Iceberg 1.9.2, Avro 1.12.0; source written with Delta 4.2.0 / Spark 4.1.1, deletion vectors disabled; local path-based tables). ```sql CREATE TABLE delta_rename (id INT, country STRING, name STRING) USING DELTA PARTITIONED BY (country) LOCATION '<path>' TBLPROPERTIES ('delta.enableDeletionVectors'='false'); INSERT INTO delta_rename (id, name, country) VALUES (1, 'John Smith', 'USA'), (2, 'Mary Johnson', 'USA'); -- incremental case: sync here (returns FULL); the two-row target reads correctly in all three readers ALTER TABLE delta_rename SET TBLPROPERTIES ('delta.columnMapping.mode'='name'); ALTER TABLE delta_rename RENAME COLUMN name TO new_name; INSERT INTO delta_rename (id, new_name, country) VALUES (6, 'Luca Verdi', 'Italy'); -- sync: first-ever sync returns FULL; a second sync returns INCREMENTAL ``` ```python targets = spark._jvm.java.util.ArrayList(); targets.add("ICEBERG") spec = (spark._jvm.org.apache.xtable.spark.TableSyncSpec.builder() .key("delta_rename").basePath(path).sourceFormat("DELTA") .targets(targets).useDeltaKernel(True).build()) r = spark._jvm.org.apache.xtable.spark.XTableSyncService().sync(spec, spark.sparkContext._jsc.hadoopConfiguration()) print(r.get("ICEBERG").getMode(), r.get("ICEBERG").getTableFormatSyncStatus().getStatusCode()) ``` Observed: `SUCCESS` in both modes; rows `(1, USA, NULL)`, `(2, USA, NULL)`, `(6, Italy, Luca Verdi)` in Spark and DuckDB; PyIceberg raises `Could not find field with name: name`; field ID 3 unchanged. Contrast: the pre-rename incremental baseline (before `SET TBLPROPERTIES`) is correct in all three readers; with column mapping set at creation the names are correct and the partitions are NULL (#A). ### Expected behavior The Kernel source sets `storageName` on struct fields so that `IcebergConversionTarget` adds the physical name to the name mapping, matching the Standalone source after #766; the three rows then read identically from Delta and Iceberg. A regression case in `ITConversionController` for "column mapping enabled after the initial write, then rename" on both sources would keep the two paths aligned (#903 is adding Kernel coverage there). ### Affected versions `main` 9d9f9b3 (0.5.0-SNAPSHOT): ID preserved, alias missing. `0.4.0-incubating` spark-runtime: same. `0.3.0-incubating` utilities: additionally changes the field ID (3 → 4) and loses all three names on incremental sync. Standalone source on `main` not run; by code reading it sets `storageName`. #### Attachments `02-delta-rename.zip`: `versions.json`, `matrix.json` (twelve Delta fixtures: 0.3.0 / 0.4.0 / main × early / late mapping × full / incremental), `source-evidence.json` (SQL, Delta schema with physical names, Parquet SHA-256 before and after), `consumer-results.json` (exact rows and exceptions per reader), `metadata/*.metadata.json` originals with decoded `*.schemas-and-name-mapping.json`, `physical-files-and-partitions.json` (Parquet physical names and field IDs per file). Source Parquet files unchanged; no compaction, REORG, rewrite, target recreation or metadata edit was used. ### Are you willing to submit PR? - [ ] I am willing to submit a PR! - [ ] I am willing to submit a PR but need help getting started! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
