salimTutuncu opened a new issue, #978: URL: https://github.com/apache/incubator-xtable/issues/978
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues. ### Please describe the bug 🐞 ### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues. ### Please describe the bug 🐞 A Hudi 0.15.0 COW table renamed with `hoodie.schema.on.read.enable=true` and `ALTER TABLE ... RENAME COLUMN name TO new_name` reads correctly in Hudi, but the Iceberg target generated over the unchanged Parquet files loses the two pre-rename values or fails to read them. The sync reports `SUCCESS`. First sync after the rename (returned mode `FULL`): ```text expected (Hudi source): (1, USA, John Smith) (2, USA, Mary Johnson) (6, Italy, Luca Verdi) Spark 4.1.1 / Iceberg 1.11.0: (1, USA, NULL) (2, USA, NULL) (6, Italy, Luca Verdi) PyIceberg 0.12.0: ValueError: Could not find field with name: name DuckDB 1.5.6: (1, NULL, NULL) (2, NULL, NULL) (6, Italy, Luca Verdi) ``` Sync before the rename, then again after it (returned modes `FULL`, then `INCREMENTAL`, same table UUID): all three readers return `(1, USA, NULL)`, `(2, USA, NULL)`, `(6, Italy, Luca Verdi)`; the Iceberg schema drops field 7 (`name`) and adds field 9 (`new_name`). The pre-rename baseline is correct in all three readers. Reproduced on unmodified `main` 9d9f9b3 (`0.5.0-SNAPSHOT`, Hudi reader 1.2.1), `0.4.0-incubating` (Hudi reader 0.14.1) and `0.3.0-incubating` (Hudi reader 0.14.0). The DuckDB NULL `country` in the first case is a reader difference and is reported as observed, not attributed to the rename. ### Why this is not a duplicate of #282 or #711 #282 is the opposite direction: Iceberg source, rename not carried into the Hudi / Delta targets. #711 / #712 concern a Delta source; #712 detects renames in `IcebergSchemaSync` by matching source field IDs, which the Hudi source does not provide, so it would not cover this case. The Delta counterparts from the same fixture are #B (name mapping) and #A (partitions); the root causes differ, hence a separate issue. ### Root cause `HudiSchemaExtractor.schema(Schema)` delegates to `AvroSchemaConverter.toInternalSchema`: https://github.com/apache/incubator-xtable/blob/9d9f9b3ced22adea03001b4b5e547b51fa7c8adc/xtable-core/src/main/java/org/apache/xtable/hudi/HudiSchemaExtractor.java#L53-L56 The Avro schema carries no field IDs, so `IcebergSchemaExtractor` assigns IDs itself: on the incremental sync the renamed column arrives as a new field (9) and the generated name mapping maps `name` → 7 and `new_name` → 9 instead of one field with two names. On the first-ever sync the mapping has field 7 named only `new_name`, while the two old Parquet files physically contain `name` and no field IDs, so the old values cannot be resolved. Hudi's InternalSchema (schema-on-read) carries stable column IDs that identify the rename; whether the source should consume it is a design question for the maintainers. Diagnostic, not a repair: adding `name` as an alias of field 7 in an in-memory copy of the first-sync metadata makes PyIceberg return all three rows; it does not address the 7 → 9 identity change of the incremental case. No stored metadata was modified. Complete schema history and decoded name mapping of the `main` incremental target: ```json {"current-schema-id":2, "schemas":[{"type":"struct","schema-id":0,"fields":[]}, {"type":"struct","schema-id":1,"fields":[{"id":1,"name":"_hoodie_commit_time"},{"id":2,"name":"_hoodie_commit_seqno"},{"id":3,"name":"_hoodie_record_key"},{"id":4,"name":"_hoodie_partition_path"},{"id":5,"name":"_hoodie_file_name"},{"id":6,"name":"id","type":"int"},{"id":7,"name":"name","type":"string"},{"id":8,"name":"country","type":"string"}]}, {"type":"struct","schema-id":2,"fields":[{"id":1,"name":"_hoodie_commit_time"},{"id":2,"name":"_hoodie_commit_seqno"},{"id":3,"name":"_hoodie_record_key"},{"id":4,"name":"_hoodie_partition_path"},{"id":5,"name":"_hoodie_file_name"},{"id":6,"name":"id","type":"int"},{"id":8,"name":"country","type":"string"},{"id":9,"name":"new_name","type":"string"}]}], "schema.name-mapping.default":[{"field-id":1,"names":["_hoodie_commit_time"]},{"field-id":2,"names":["_hoodie_commit_seqno"]},{"field-id":3,"names":["_hoodie_record_key"]},{"field-id":4,"names":["_hoodie_partition_path"]},{"field-id":5,"names":["_hoodie_file_name"]},{"field-id":6,"names":["id"]},{"field-id":7,"names":["name"]},{"field-id":8,"names":["country"]},{"field-id":9,"names":["new_name"]}]} ``` (Field types omitted for the `_hoodie_*` string columns for brevity; the full JSON is attached.) ### To Reproduce Environment: sync host Spark 3.5.5 / JDK 17; XTable `main` 9d9f9b3 built with Java 11 (artifact SHA-256 `f863a05924b31230bf4ba2a655e7e0712b5d96c91dfcdf1171fba5c751759fc6`); classpath: the built `xtable-spark-runtime_2.12-0.5.0-SNAPSHOT.jar`, `hudi-spark3.4-bundle_2.12-1.2.1.jar` (main's declared Hudi reader), `iceberg-spark-runtime-3.5_2.12-1.9.2`, `avro-1.12.0`, `jol-core-0.16`, `spark.driver.userClassPathFirst=true`. Source written and read with Hudi 0.15.0 / Spark 3.5.5 (Hudi 1.2.1 is only the metadata reader inside XTable). Local path-based tables; timeline server and metadata table disabled to keep the fixture small. ```sql CREATE TABLE hudi_rename (id INT, country STRING, name STRING) USING HUDI PARTITIONED BY (country) LOCATION '<path>' OPTIONS ('type'='cow', 'primaryKey'='id', 'preCombineField'='id', 'hoodie.metadata.enable'='false', 'hoodie.clean.automatic'='false', 'hoodie.embed.timeline.server'='false', 'hoodie.parquet.small.file.limit'='0'); INSERT INTO hudi_rename (id, name, country) VALUES (1, 'John Smith', 'USA'), (2, 'Mary Johnson', 'USA'); -- incremental case: sync here (returns FULL) SET hoodie.schema.on.read.enable=true; ALTER TABLE hudi_rename RENAME COLUMN name TO new_name; INSERT INTO hudi_rename (id, new_name, country) VALUES (6, 'Luca Verdi', 'Italy'); -- sync: first-ever sync returns FULL; a second sync returns INCREMENTAL ``` ```python targets = spark._jvm.java.util.ArrayList(); targets.add("ICEBERG") spec = (spark._jvm.org.apache.xtable.spark.TableSyncSpec.builder() .key("hudi_rename").basePath(path).sourceFormat("HUDI") .targets(targets).partitionSpec("country:VALUE").build()) r = spark._jvm.org.apache.xtable.spark.XTableSyncService().sync(spec, spark.sparkContext._jsc.hadoopConfiguration()) print(r.get("ICEBERG").getMode(), r.get("ICEBERG").getTableFormatSyncStatus().getStatusCode()) ``` Observed: `SUCCESS` in both cases with the rows shown above; `name` → 7 and `new_name` → 9 after the incremental sync. Contrast: `SELECT * FROM hudi_rename` in the Hudi source session returns the three expected rows after the rename, and the pre-rename incremental baseline reads correctly from Iceberg in all three readers. ### Expected behavior A Hudi schema-on-read rename preserved as an Iceberg rename (same field ID) with the old physical name kept as a name-mapping alias, so the pre-rename files remain readable. If Hudi's column IDs are intentionally not consumed, a documented limitation plus a sync-time warning when a Hudi source schema drops and adds a column of the same type would at least replace the silent `SUCCESS`. ### Affected versions `main` 9d9f9b3 (Hudi reader 1.2.1), `0.4.0-incubating` spark-runtime (Hudi reader 0.14.1), `0.3.0-incubating` utilities (Hudi reader 0.14.0): same per-case results. Source tested: Hudi 0.15.0 COW via Spark SQL only; no claim about other Hudi versions, MOR, or other evolution operations. ### Are you willing to submit PR? - [ ] I am willing to submit a PR! - [ ] I am willing to submit a PR but need help getting started! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
