JingsongLi commented on code in PR #9347:
URL: https://github.com/apache/paimon/pull/9347#discussion_r3888247193
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaManager.java:
##########
@@ -217,6 +219,18 @@ public TableSchema createTable(Schema schema, boolean
externalTable) throws Exce
}
}
+ /**
+ * Shift all field ids of a new table by one when {@link
CoreOptions#FIELD_ID_ONE_BASED} is set.
+ * Applied only at table creation: data files embed these ids (Parquet
footers, Iceberg
+ * metadata), so the id space of an existing table must never be re-based.
+ */
+ private static Schema applyFieldIdOneBased(Schema schema) {
+ if (!CoreOptions.fromMap(schema.options()).fieldIdOneBased()) {
+ return schema;
+ }
+ return schema.copy((RowType) ShiftFieldId.shift(schema.rowType(), 1));
Review Comment:
[P1] Preserve IDs when the input schema is already resolved
This unconditionally shifts every schema carrying `field-id.one-based=true`,
but some create paths pass a persisted `TableSchema` back into table creation.
In particular, Spark `CopySchemaOperator.newSchemaFromTableSchema` copies both
the already-shifted fields and this option, so `copy_files` turns source IDs
`[1, 2, 5]` into target IDs `[2, 3, 6]`. The copied Parquet files still contain
the original IDs, leaving the target Iceberg metadata inconsistent with the
physical files. Please make this transformation idempotent or provide an
explicit path that preserves already-assigned IDs, and cover one-based
`copy_files` with a regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
