vinishjail97 opened a new issue, #871: URL: https://github.com/apache/incubator-xtable/issues/871
Reported by https://github.com/kvutukuru on the [ASF XTable Slack](https://apachextable.slack.com/archives/C089PRPRZ60/p1785168836619579). GitHub issue creation was blocked on their side, so filing on their behalf. ## Summary When XTable creates a new Iceberg target for a partitioned source, the resulting table ends up with two partition specs: an unused `spec-id 0` (unpartitioned) and `spec-id 1` carrying the real source partitioning. All data files are written under spec 1, and no data ever existed under spec 0, but the retained spec 0 makes Iceberg-consuming engines classify the table as having undergone partition evolution. Databricks rejects such tables on its Iceberg-to-Delta path: ``` [DELTA_CLONE_INCOMPATIBLE_SOURCE.ICEBERG_UNDERGONE_PARTITION_EVOLUTION] The clone source has valid format, but has unsupported feature with Delta. Source iceberg table has undergone partition evolution. SQLSTATE: 0AKDC ``` ## Environment - XTable: `xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar` (behavior unchanged on `main`, see code pointers below) - Source: Apache Hudi, COPY_ON_WRITE, Parquet, partitioned by `df_meta_p_cppib` - Target: Iceberg format version 2, AWS Glue catalog, S3FileIO - Consumers: AWS Athena (Trino) and Spark on EMR read the table fine; Databricks fails ## Observed metadata A single XTable run produces three metadata files within ~2 seconds, plus one snapshot: ``` 00000-444ce851-....metadata.json 00001-87e5211c-....metadata.json 00002-312c2a3b-....metadata.json snap-5065718056232190869-1-2b42ae3a-....avro 2b42ae3a-...-m0.avro ``` The current metadata contains: ```json { "format-version": 2, "default-spec-id": 1, "last-partition-id": 1000, "partition-specs": [ { "spec-id": 0, "fields": [] }, { "spec-id": 1, "fields": [ { "name": "df_meta_p_cppib", "transform": "identity", "source-id": 14, "field-id": 1000 } ]} ] } ``` Diffing the spec-related fields of `00001` against `00002` produces no output, so the spec is added between `00000` and `00001`, before any data snapshot exists. The manifest confirms all data uses `partition_spec_id = 1`, one snapshot, operation `append`, 1 data file, 260 rows. ## Root cause `IcebergTableManager.getOrCreateTable` (`xtable-core/src/main/java/org/apache/xtable/iceberg/IcebergTableManager.java:67-110`) deliberately creates the table with an empty schema and an unpartitioned spec, then commits the real schema and spec in a second metadata commit so that Iceberg does not remap source field IDs: ```java // initialize the table with an empty schema, then manually set the schema to prevent the // Iceberg API from remapping the field IDs. catalog.createTable( tableIdentifier, new Schema(), PartitionSpec.unpartitioned(), basePath, Collections.emptyMap()); ... TableMetadata.Builder builder = TableMetadata.buildFrom(tableMetadata); builder.setCurrentSchema(schema, schema.highestFieldId()); builder.setDefaultPartitionSpec(partitionSpec); operations.commit(tableMetadata, builder.build()); ``` `createTable` commits metadata that already contains `spec-id 0 = unpartitioned`. The follow-up `setDefaultPartitionSpec(partitionSpec)` cannot replace that spec, so Iceberg appends the real spec as `spec-id 1` and moves `default-spec-id` to 1. Spec 0 stays in `partition-specs` for the life of the table. The `HadoopTables` branch a few lines below has the same shape. The empty-schema initialization is intentional and needed for field-ID preservation; the leftover unpartitioned spec looks like an unintended side effect of it rather than a design goal. ## Impact The table is spec-correct and readable by Iceberg-native engines, but any consumer that requires "no partition evolution" rejects it. That includes Databricks CONVERT TO DELTA / CLONE from Iceberg, which is the path Databricks uses to expose an external Iceberg table. So a Hudi table synced to Iceberg by XTable is not consumable from Databricks even though the partitioning never actually changed. ## Expected behavior For a newly created target, the intended partitioning should land as `spec-id 0` in the first metadata the table ever exposes: ```json { "default-spec-id": 0, "partition-specs": [ { "spec-id": 0, "fields": [ { "name": "df_meta_p_cppib", "transform": "identity", "source-id": 14, "field-id": 1000 } ]} ] } ``` with `table.specs()` of size 1 and all manifests at `partition_spec_id = 0`. ## Proposed direction The naive fix of passing the real spec straight to `catalog.createTable` does not work on its own: the spec has to be bound to a schema, and the whole point of the empty-schema create is to stop Iceberg from reassigning field IDs. Two options that preserve field IDs: 1. Keep `createTable` with the empty schema purely to register the table in the catalog and obtain `TableOperations`, but build the committed metadata with `TableMetadata.buildFromEmpty()` (public in Iceberg 1.9.2) instead of `buildFrom(current)`, carrying over UUID, location, format version and properties, then `setCurrentSchema(schema, schema.highestFieldId())` and `setDefaultPartitionSpec(partitionSpec)`. With an empty spec list, the real spec is assigned `INITIAL_SPEC_ID` (0), so the table's first visible metadata has exactly one spec. Field IDs are still set explicitly, so no remapping. 2. `TableMetadata.Builder.removeSpecs(...)` would express the intent directly, but it is package-private in `org.apache.iceberg` and therefore not callable from `org.apache.xtable.iceberg` without an accessor class in Iceberg's package. Option 1 looks like the smaller change. Happy to hear if maintainers prefer a different route, or a create-with-final-spec mode gated behind a config rather than a default change. Note on scope: any fix only helps newly created targets. Tables already carrying the extra spec 0 need the Iceberg metadata recreated to become Databricks-compatible. ## Suggested test After an initial sync of a partitioned Hudi source, and again after a second sync: ```java assertThat(table.specs()).hasSize(1); assertThat(table.spec().specId()).isEqualTo(0); assertThat(table.spec().fields()).hasSize(1); assertThat(table.spec().fields().get(0).name()).isEqualTo("df_meta_p_cppib"); ``` ## Reproduction 1. Create a Hudi COW table partitioned by `df_meta_p_cppib`. 2. Run XTable against the Hudi base path with an Iceberg target in AWS Glue. 3. Inspect the current metadata JSON: ``` jq '{default_spec_id: ."default-spec-id", partition_spec_count: (."partition-specs" | length), partition_specs: ."partition-specs"}' current.metadata.json ``` 4. Observe `spec 0 = unpartitioned`, `spec 1 = identity(df_meta_p_cppib)`. 5. Confirm only spec 1 is used: ```sql SELECT partition_spec_id, COUNT(*) FROM "<db>"."<table>$manifests" GROUP BY partition_spec_id; ``` 6. Attempt to consume the table from Databricks and observe `DELTA_CLONE_INCOMPATIBLE_SOURCE.ICEBERG_UNDERGONE_PARTITION_EVOLUTION`. ## Secondary observation (possibly a separate issue) Pre-creating the Iceberg table externally, so the intended partitioning is already spec 0 before XTable syncs, was attempted as a workaround. The Iceberg version bundled in the XTable snapshot fails to parse metadata containing explicit nulls for `initial-default` / `write-default`: ``` java.lang.NullPointerException: Cannot create expression literal from null at org.apache.iceberg.expressions.Literals.from at org.apache.iceberg.expressions.Expressions.lit at org.apache.iceberg.SchemaParser.defaultFromJson ``` This makes the pre-create workaround impractical. Filing it here as context; happy to split it out if maintainers prefer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
