vbabenkoru opened a new pull request, #9347:
URL: https://github.com/apache/paimon/pull/9347
### Purpose
Paimon assigns field ids starting from 0, while the Iceberg ecosystem
assigns them starting from 1. Iceberg metadata produced for a Paimon table
therefore contains a field with id 0, which some external Iceberg readers
reject outright — Snowflake, for example, refuses such tables — and which
forces workarounds elsewhere (the REST committer creates a dummy schema, or
falls back to partition evolution, when a partition column has field id 0).
This PR adds an opt-in table option, `field-id.one-based`, that shifts every
field id — top-level and nested — by one at table creation, so the table's id
space matches what Iceberg-native readers expect from the start.
```
CREATE TABLE t (...) WITH (
'field-id.one-based' = 'true',
'metadata.iceberg.storage' = 'rest-catalog',
...
);
```
Design points:
* **Creation-time only, immutable.** Field ids are embedded in data files
(Parquet footers) and Iceberg metadata; re-basing an existing table would
corrupt column resolution. The option is `@Immutable` and applied once in
`SchemaManager.createTable`; existing tables keep their ids.
* **A shift, not a reassignment.** The new `ShiftFieldId` visitor translates
the whole id space by a fixed offset, preserving relative order and any gaps,
unlike `ReassignFieldId` which renumbers densely. Schema evolution then
continues naturally from the shifted highest field id.
* **Default unchanged.** With the option unset, ids remain zero-based,
byte-for-byte identical to today.
### Tests
* `FieldIdOneBasedTest` (paimon-core): default stays zero-based; the shift
applies to nested ids; schema evolution continues from shifted ids; the option
is create-time-only and immutable.
* `IcebergFieldIdOneBasedCompatibilityTest` (paimon-iceberg): end-to-end on
a primary-key DV table with Iceberg format version 3 — Paimon schema, Iceberg
metadata, and Parquet footers all carry the same strictly positive ids,
verified against the Iceberg reader; plus a control test that the default
remains zero-based.
### API and Format
New optional table option `field-id.one-based` (default `false`, immutable).
No change to existing tables or to the default id assignment.
### Documentation
Option is documented via its description (generated configuration docs).
---
*AI notice: The code is generated using Fable 5 (with reviews from Codex)
but has been verified to run on a real cluster with Flink, Paimon, Iceberg,
StarRocks and Snowflake.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
