GitHub user nevzheng created a discussion: [DISCUSS] Iceberg V3 type support in Gravitino
### Current state While working on #11927 / #11928 (the native API failing to load Iceberg tables with a V3 `variant` column), I found that Gravitino's unified type model has no representation for Iceberg's V3 net-new types. With #11928 merged, all of them load through the native API as `ExternalType`: | Iceberg V3 type | Maps to today | Native Gravitino type? | |---|---|---| | `variant` | `external(variant)` | none | | `geometry` | `external(GEOMETRY)` | none | | `geography` | `external(GEOGRAPHY)` | none | | `timestamp_ns` | `external(TIMESTAMP_NANO)` | looks possible — `TimestampType` takes a precision | | `timestamptz_ns` | `external(TIMESTAMP_NANO)` | looks possible — `TimestampType` takes a precision | | `unknown` | `external(UNKNOWN)` | maybe `NullType` — defer to maintainers | ### The question Is this working-as-intended, or do we want native support for these types — and if so, how? ### Decisions we need 1. Is `ExternalType` the intended long-term mapping for Iceberg V3 types, or is this a gap to close? 2. If we want native support — which types, and how do they enter the unified model (enum + JSON serde + DTOs + clients + per-catalog converters)? ### Options to seed the discussion - **A — Keep V3 types as `ExternalType`.** Document the limitations, defer native support. This is what `ExternalType` was designed for (#3500). But note the downstream cost: `external(...)` columns aren't always consumable — e.g. `external(serial)` failed when queried through Trino (#10957), and the fix there was to give the type a *native* mapping rather than leave it external. - **B — Implement native support for V3 types.** Add native types (e.g. `VariantType`, geospatial types) to the unified model. These are independent types (variant, geospatial, nanosecond timestamps), so they could be pursued separately / incrementally rather than as one change. `timestamp_ns` in particular looks close — `TimestampType` already carries precision (#2431). ### References **How the escape hatches were designed** - #3500 / #3501 — `ExternalType` added for "unsolved catalog type" (its intended purpose). - #2117 / #2140 — `UnparsedType` added to handle an unresolvable type from the catalog (the alternative fallback). **Where `ExternalType` has caused problems** - #10957 / #10970 — `external(serial)` unqueryable via Trino; resolved by mapping to a native type. - #11805 — `ExternalType.catalogString()` written verbatim into DDL SQL (write-path hazard). - #11910 / #11915 — ClickHouse types falling through to `ExternalType`. **Iceberg V3 surface** - #11927 / #11928 — the `variant` load bug + fix that surfaced this. - #10994 — variant on the IRC/JDBC backend; noted as under a dedicated community meeting. - #11293 / #11219 — server-side Iceberg upgraded to 1.11.0 (brought V3 types into range). - #10995, #10986 — other V3 gaps (deletion vectors; format-version-3 downgraded to 2). **Timestamp precision precedent** - #2431 — time/timestamp precision support (why `timestamp_ns` looks representable natively). GitHub link: https://github.com/apache/gravitino/discussions/11929 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
