GitHub user nevzheng edited a comment on the discussion: [DISCUSS] Iceberg V3
type support in Gravitino
# Proposal: Native `variant` type support in Gravitino
Status: draft / notes
## Summary
Promote Iceberg V3 `variant` from the `ExternalType("variant")` stopgap
(#11928) to a
first-class `VariantType` leaf. It stays opaque and just gains a real identity
in the
type model. Small change: at the schema level a variant is a parameterless
leaf, the
same shape as the existing `UUIDType`.
## What a variant is
A column type for arbitrary, JSON-like semi-structured data whose shape varies
row to
row. At the schema level it is the bare, parameterless word `variant`: all
structure
lives in the values, not the type. A metadata catalog sees one opaque leaf.
## Current state
Gravitino only records that a column is a variant; it never reads values. The
IRC path
already handled it (#10994); the native converter (#11927) threw
`UnsupportedOperationException:
Unsupported type: variant`. After #11928 it maps to
`ExternalType.of("variant")`, so the
column loads. That is already a relabeled blob, but as a string, not a type:
nothing can
branch on it or map it deliberately per connector.
## The change
Serde here is of the schema, not values: a `Type` object round-trips to JSON to
cross the
REST API and the entity store. A variant leaf serializes to the bare token
`"variant"`
(like `uuid`), so this is four small edits:
- `Type.Name.VARIANT` - new enum tag.
- `Types.VariantType` - singleton leaf extending `PrimitiveType` (a `UUIDType`
clone).
- `JsonUtils` - add `VariantType.get()` to the primitive `TYPES` map (read) and
a
`case VARIANT` in `writeDataType` (write).
- `FromIcebergType.variant()` - return `Types.VariantType.get()` instead of
`ExternalType`.
Out of scope: introspection (no sub-fields, no `variant_get`-style accessors).
Those are
query-engine operations on row data, not catalog metadata.
## Decisions and trade-offs
- **Extend `PrimitiveType`** (not `ComplexType`, not raw `Type`): a
parameterless childless
leaf fits the primitive serde path with no bespoke code. Risk: "primitive"
reads oddly
for semi-structured data, but the base class classifies by structure and
`uuid`/`binary`
set the precedent.
- **Iceberg-first**: other connectors keep mapping variant to `ExternalType`.
Adding
`Name.VARIANT` touches every `Type.Name` switch (Hive, Glue, Paimon, Lance,
Trino,
Flink, JDBC, Spark), but only Iceberg emits one today. Risk: a new constant
can reach a
switch `default` elsewhere; audit each and convert connectors individually
later.
- **Why not stay external**: `external(serial)` is unqueryable via Trino
(#10957) and
`catalogString()` has been written verbatim into DDL SQL unvalidated
(#11805). A modeled
identity avoids passing an opaque string through those paths.
## PoC
https://github.com/apache/gravitino/pull/11932
## References
- #11927 (open, bug) - native metadata API fails to load V3 variant columns.
- #11928 (open, PR) - the `ExternalType` stopgap this builds on.
- #10994 (closed) - V3 variant in the Iceberg REST server JDBC backend (the IRC
path).
- #3500 (closed) - `ExternalType` for unsolved catalog type (the escape hatch's
origin).
- #10957 (open) - `external(serial)` unqueryable via Trino.
- #11805 (open) - `ExternalType.catalogString()` written verbatim into DDL SQL.
GitHub link:
https://github.com/apache/gravitino/discussions/11929#discussioncomment-17567285
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
