nevzheng opened a new pull request, #11969:
URL: https://github.com/apache/gravitino/pull/11969
### What changes were proposed in this pull request?
Map Iceberg V3's `unknown` type to Gravitino's existing `Types.NullType`, so
an `unknown` column loads through the native metadata API as a first-class
`null` type instead of the `ExternalType("UNKNOWN")` stopgap it resolves to
today.
- `catalog-lakehouse-iceberg`: `FromIcebergType` maps `unknown → NullType`;
`ToIcebergType` / `ToIcebergTypeVisitor` map `NullType → unknown` (via a
`nullType()` dispatch hook, since `NullType` isn't a `PrimitiveType`), and
reject a required (non-nullable) `unknown` column per the Iceberg spec.
- docs: unified type reference (`Null type` section), Iceberg type-mapping
table, OpenAPI examples.
No new `api` type and no `common`/Python serde change — `NullType` already
exists, round-trips as the JSON token `"null"`, and is already mapped for
Spark, Flink, Lance, and CLI (the Python client decodes `"null"` to `NullType`
with no change).
Builds on the merged native `variant` support (#11932) and format-version-3
support (#11954).
Fixes #11951
### Why are the changes needed?
Loading an Iceberg V3 table with an `unknown` column through the native API
resolves it to `ExternalType("UNKNOWN")` — opaque (nothing can branch on it),
not writable back, and external types have caused downstream problems
(unqueryable via Trino #10957; `catalogString()` written verbatim into DDL
#11805). `unknown` is the universal null/void column type — Iceberg's own
converters map it to each engine's null type (Spark `NullType`, Flink `NULL`,
Arrow `null`) — and Gravitino already models that as `NullType`, so this is
just the missing wiring.
### Does this PR introduce _any_ user-facing change?
Yes — Iceberg V3 tables with an `unknown` column now load through the native
API as `null` (previously `external(UNKNOWN)`), and a `null`-typed column can
be written to a format-version-3 Iceberg table. Other connectors are unchanged.
Connector propagation (reject-with-test for engines without a null-type
equivalent) is a planned follow-up.
### How was this patch tested?
- Unit (`TestConvertUtil`): `testUnknownType` (converter both directions)
and `testUnknownColumnToIcebergSchema` (write path → optional `unknown` field;
required column rejected).
- Docker IT (`CatalogIcebergRestIT`, passing) — both cross-surface
directions between the Iceberg REST (IRC) API and the native metadata API:
- `testV3TypeConversionViaIcebergClient`: IRC writes an `unknown` column →
the native API reads it back as `NullType`.
- `testCreateUnknownColumnWriteRoundTrip`: the native API writes a
`NullType` column at format-version 3 → native reload returns `NullType`, and
the IRC reads the same table back as Iceberg `unknown`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
