nevzheng opened a new issue, #11951:
URL: https://github.com/apache/gravitino/issues/11951
### Describe the feature
Support Iceberg V3's `unknown` type natively by mapping it to Gravitino's
existing `Types.NullType`, so `unknown` columns load through the native
metadata API as a first-class `null` type rather than the
`ExternalType("UNKNOWN")` stopgap they resolve to today.
Following the native `variant` work (#11927 / #11932 / #11949), this is the
sibling change for the next V3 net-new type. Unlike `variant`, `unknown`
already has a natural home in Gravitino's type model — `NullType` — so no new
`api` type is needed.
### Motivation
Iceberg V3 adds `unknown`: a placeholder column type for data whose type
isn't known yet — optional, null-defaulted, not stored in data files, and
promotable to any concrete type via schema evolution
([spec](https://iceberg.apache.org/spec/)).
Today an `unknown` column loads through the native API as
`ExternalType("UNKNOWN")`. It loads (unlike `variant`, which threw — #11927),
but `external(UNKNOWN)` is opaque: nothing can branch on it, and external types
have caused downstream problems (unqueryable via Trino #10957;
`catalogString()` written verbatim into DDL #11805).
`unknown` is not an Iceberg-specific concept — it is the universal "null /
void" column type. Iceberg's own converters map it to each engine's null type
(`⇄ Spark NullType`, `⇄ Flink NULL`, `⇄ Arrow null`), and Gravitino already
models this as `Types.NullType`, mapped bidirectionally for Spark, Flink,
Lance, CLI, and JSON. Iceberg is simply the missing wiring.
### Example usages
- A Spark job writing an all-null / untyped column (`SELECT NULL AS c`, or
an all-null Parquet source) produces an Iceberg V3 table with an `unknown`
column. Loading it through Gravitino's native API today yields
`external(UNKNOWN)`; with this change it yields `null`.
- Defining a placeholder column now and promoting it to a concrete type
later via schema evolution — Gravitino shows `null` while unknown, then the
promoted type (e.g. `string`) on the next load.
### Semantics
A Gravitino `null`-typed column (Iceberg `unknown`) holds only null values —
engines must not populate or fill it — and may be repurposed to a concrete type
later via schema evolution, as expected of the Iceberg `unknown` placeholder.
This matches how Spark/Flink/Arrow already surface it as their `null`/`void`
type; the "not yet typed" state is transient and self-corrects on the next load
once the column is promoted.
### Proposed approach
Map Iceberg `unknown` ⇄ Gravitino `Types.NullType` in the Iceberg catalog
converter, with any Iceberg-specific constraints (e.g. optional column)
enforced locally in the Iceberg connector. Reusing the existing `NullType`
keeps the surface small — no new `api` type or serde. Likely to land as a
stack, with broader connector propagation handled as follow-ups.
### Additional context
- `Types.NullType` already exists and is bidirectional for Spark, Flink,
Lance, CLI, and JSON; it's just not yet wired into Iceberg (or documented in
the type reference).
- Considered minting a new native `UnknownType` mirroring `variant`
(#11932), but `unknown` already has an exact equivalent in `NullType` and is a
universal concept — reusing it avoids duplicating a type across every
connector. Noted so maintainers can weigh in.
- Related: #11927 (V3 load bug), #11932 / #11949 (native `variant`), #10957
/ #11805 (external-type hazards).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
