This is an automated email from the ASF dual-hosted git repository.
mchades pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 69ef06b81b [#11951] feat(iceberg): map V3 unknown type to Gravitino
NullType (#11969)
69ef06b81b is described below
commit 69ef06b81ba947a275922086a720a92335d644ad
Author: Nevin Zheng <[email protected]>
AuthorDate: Tue Jul 14 20:24:37 2026 -0700
[#11951] feat(iceberg): map V3 unknown type to Gravitino NullType (#11969)
### What changes were proposed in this pull request?
Map Iceberg V3's `unknown` type to Gravitino's existing
`Types.NullType`, so an `unknown` column loads through the native
metadata API as a first-class `null` type instead of the
`ExternalType("UNKNOWN")` stopgap it resolves to today.
- `catalog-lakehouse-iceberg`: `FromIcebergType` maps `unknown →
NullType`; `ToIcebergType` / `ToIcebergTypeVisitor` map `NullType →
unknown` (via a `nullType()` dispatch hook, since `NullType` isn't a
`PrimitiveType`), and reject a required (non-nullable) `unknown` column
per the Iceberg spec.
- docs: unified type reference (`Null type` section), Iceberg
type-mapping table, OpenAPI examples.
No new `api` type and no `common`/Python serde change — `NullType`
already exists, round-trips as the JSON token `"null"`, and is already
mapped for Spark, Flink, Lance, and CLI (the Python client decodes
`"null"` to `NullType` with no change).
Builds on the merged native `variant` support (#11932) and
format-version-3 support (#11954).
Fixes #11951
### Why are the changes needed?
Loading an Iceberg V3 table with an `unknown` column through the native
API resolves it to `ExternalType("UNKNOWN")` — opaque (nothing can
branch on it), not writable back, and external types have caused
downstream problems (unqueryable via Trino #10957; `catalogString()`
written verbatim into DDL #11805). `unknown` is the universal null/void
column type — Iceberg's own converters map it to each engine's null type
(Spark `NullType`, Flink `NULL`, Arrow `null`) — and Gravitino already
models that as `NullType`, so this is just the missing wiring.
### Does this PR introduce _any_ user-facing change?
Yes — Iceberg V3 tables with an `unknown` column now load through the
native API as `null` (previously `external(UNKNOWN)`), and a
`null`-typed column can be written to a format-version-3 Iceberg table.
Other connectors are unchanged. Connector propagation (reject-with-test
for engines without a null-type equivalent) is a planned follow-up.
### How was this patch tested?
- Unit (`TestConvertUtil`): `testUnknownType` (converter both
directions) and `testUnknownColumnToIcebergSchema` (write path →
optional `unknown` field; required column rejected).
- Docker IT (`CatalogIcebergRestIT`, passing) — both cross-surface
directions between the Iceberg REST (IRC) API and the native metadata
API:
- `testV3TypeConversionViaIcebergClient`: IRC writes an `unknown` column
→ the native API reads it back as `NullType`.
- `testCreateUnknownColumnWriteRoundTrip`: the native API writes a
`NullType` column at format-version 3 → native reload returns
`NullType`, and the IRC reads the same table back as Iceberg `unknown`.
---
.../iceberg/converter/FromIcebergType.java | 4 ++
.../lakehouse/iceberg/converter/ToIcebergType.java | 14 +++++
.../iceberg/converter/ToIcebergTypeVisitor.java | 7 +++
.../iceberg/converter/TestConvertUtil.java | 43 ++++++++++++++
.../integration/test/CatalogIcebergBaseIT.java | 67 ++++++++++++++++++++--
docs/lakehouse-iceberg-catalog.md | 8 +++
docs/manage-relational-metadata-using-gravitino.md | 13 +++++
docs/open-api/datatype.yaml | 5 +-
8 files changed, 153 insertions(+), 8 deletions(-)
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/FromIcebergType.java
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/FromIcebergType.java
index 1cf732e0e7..7328ff2083 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/FromIcebergType.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/FromIcebergType.java
@@ -130,6 +130,10 @@ public class FromIcebergType extends
TypeUtil.SchemaVisitor<Type> {
Types.DecimalType decimal = (Types.DecimalType) primitive;
return org.apache.gravitino.rel.types.Types.DecimalType.of(
decimal.precision(), decimal.scale());
+ case UNKNOWN:
+ // Iceberg V3 unknown is the null-only placeholder type; map it to
Gravitino's NullType,
+ // matching how Iceberg's own engine converters map unknown <-> the
engine null type.
+ return org.apache.gravitino.rel.types.Types.NullType.get();
case GEOMETRY:
return org.apache.gravitino.rel.types.Types.GeometryType.of(
((Types.GeometryType) primitive).crs());
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergType.java
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergType.java
index 7f671631eb..53dfdba639 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergType.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergType.java
@@ -68,6 +68,14 @@ public class ToIcebergType extends
ToIcebergTypeVisitor<Type> {
String doc = field.comment();
+ if (type instanceof Types.UnknownType && !field.nullable()) {
+ // Iceberg requires unknown (null) columns to be optional: they are
not stored in data
+ // files and must default to null. Reject a required unknown column
with a clear message.
+ throw new IllegalArgumentException(
+ String.format(
+ "Iceberg unknown/null type column '%s' must be optional
(nullable)", field.name()));
+ }
+
if (field.nullable()) {
newFields.add(Types.NestedField.optional(id, field.name(), type, doc));
} else {
@@ -188,4 +196,10 @@ public class ToIcebergType extends
ToIcebergTypeVisitor<Type> {
}
throw new UnsupportedOperationException("Not a supported type: " +
primitive.toString());
}
+
+ @Override
+ public Type nullType(org.apache.gravitino.rel.types.Types.NullType nullType)
{
+ // Gravitino NullType maps to Iceberg's V3 unknown type (the null-only
placeholder).
+ return Types.UnknownType.get();
+ }
}
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergTypeVisitor.java
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergTypeVisitor.java
index 8d63728dea..28a6849fc9 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergTypeVisitor.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergTypeVisitor.java
@@ -53,6 +53,9 @@ public class ToIcebergTypeVisitor<T> {
fieldResults.add(visitor.field(field, visit(field.type(), visitor)));
}
return visitor.struct((Types.StructType) type, fieldResults);
+ } else if (type instanceof Types.NullType) {
+ // NullType implements Type directly rather than PrimitiveType, so it
needs its own dispatch.
+ return visitor.nullType((Types.NullType) type);
} else {
return visitor.atomic((Type.PrimitiveType) type);
}
@@ -81,4 +84,8 @@ public class ToIcebergTypeVisitor<T> {
public T atomic(Type.PrimitiveType primitive) {
throw new UnsupportedOperationException();
}
+
+ public T nullType(Types.NullType nullType) {
+ throw new UnsupportedOperationException();
+ }
}
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/TestConvertUtil.java
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/TestConvertUtil.java
index 06e6b56886..32fd08d21f 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/TestConvertUtil.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/TestConvertUtil.java
@@ -59,6 +59,49 @@ public class TestConvertUtil extends TestBaseConvert {
instanceof Types.VariantType);
}
+ @Test
+ public void testUnknownType() {
+ // Iceberg V3 unknown <-> Gravitino NullType, in both directions.
+ Assertions.assertTrue(
+ CONVERTER.toGravitino(Types.UnknownType.get())
+ instanceof org.apache.gravitino.rel.types.Types.NullType);
+ Assertions.assertTrue(
+
CONVERTER.fromGravitino(org.apache.gravitino.rel.types.Types.NullType.get())
+ instanceof Types.UnknownType);
+ }
+
+ @Test
+ public void testUnknownColumnToIcebergSchema() {
+ // Write path: a nullable null-typed column becomes an optional Iceberg
unknown field
+ // (exercises ConvertUtil.toIcebergSchema -> ToIcebergType.struct and the
nullType dispatch).
+ Column nullableColumn =
+ IcebergColumn.builder()
+ .withName("c_unknown")
+ .withType(org.apache.gravitino.rel.types.Types.NullType.get())
+ .withNullable(true)
+ .withComment(TEST_COMMENT)
+ .build();
+ Schema schema = ConvertUtil.toIcebergSchema(new Column[] {nullableColumn});
+ Types.NestedField field = schema.findField("c_unknown");
+ Assertions.assertNotNull(field);
+ Assertions.assertTrue(field.isOptional());
+ Assertions.assertTrue(field.type() instanceof Types.UnknownType);
+
+ // A required null-typed column is rejected: Iceberg unknown must be
optional.
+ Column requiredColumn =
+ IcebergColumn.builder()
+ .withName("c_unknown_required")
+ .withType(org.apache.gravitino.rel.types.Types.NullType.get())
+ .withNullable(false)
+ .withComment(TEST_COMMENT)
+ .build();
+ IllegalArgumentException exception =
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () -> ConvertUtil.toIcebergSchema(new Column[] {requiredColumn}));
+ Assertions.assertTrue(exception.getMessage().contains("must be optional"));
+ }
+
@Test
public void testGeometryType() {
// Iceberg V3 geometry <-> Gravitino GeometryType, preserving the CRS in
both directions.
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergBaseIT.java
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergBaseIT.java
index 7b465f9cca..b44bea50c7 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergBaseIT.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergBaseIT.java
@@ -727,9 +727,10 @@ public abstract class CatalogIcebergBaseIT extends BaseIT {
// like Spark 4 would, directly through the Iceberg catalog, then load
through the native
// metadata interface.
//
- // Variant, geometry, and geography load as their native Gravitino types.
Nanosecond timestamps
- // are asserted below; unknown remains an ExternalType until it gains
unified-model support.
+ // Variant, unknown, geometry, and geography load as their native
Gravitino types. Nanosecond
+ // timestamps are asserted below.
assertV3LoadsAsVariant("v3_variant");
+ assertV3LoadsAsNull("v3_unknown");
assertV3LoadsAsGeometry("v3_geometry",
org.apache.iceberg.types.Types.GeometryType.crs84());
assertV3LoadsAsGeometry(
"v3_geometry_srid",
org.apache.iceberg.types.Types.GeometryType.of("srid:3857"));
@@ -750,10 +751,6 @@ public abstract class CatalogIcebergBaseIT extends BaseIT {
"v3_timestamptz_ns",
org.apache.iceberg.types.Types.TimestampNanoType.withZone(),
Types.TimestampType.withTimeZone(NANO_PRECISION));
-
- // TODO(apache/gravitino#11929): expect native types once these gain
unified-model support.
- assertV3LoadsAsExternal(
- "v3_unknown", org.apache.iceberg.types.Types.UnknownType.get(),
"UNKNOWN");
}
@Test
@@ -818,6 +815,57 @@ public abstract class CatalogIcebergBaseIT extends BaseIT {
"3",
loaded.properties().get(IcebergTablePropertiesMetadata.FORMAT_VERSION));
}
+ @Test
+ void testCreateUnknownColumnWriteRoundTrip() {
+ // Write path: create a null-typed (Iceberg unknown) column *through the
Gravitino relational
+ // API*, write it to the REST (IRC) backend, then load it back and confirm
it round-trips as
+ // NullType. REST-backend only, for the same reason as
testV3TypeConversionViaIcebergClient:
+ // the CI Hive metastore cannot store V3 column types.
+ Assumptions.assumeTrue(
+ "rest".equalsIgnoreCase(TYPE),
+ "Unknown columns require a backend that does not validate against the
Hive metastore");
+
+ NameIdentifier ident = NameIdentifier.of(schemaName, "t_unknown_write");
+ // A null-typed column maps to Iceberg's V3 unknown type. Column.of(..)
defaults to nullable,
+ // which unknown requires (a required unknown column is rejected before
reaching the backend).
+ Column[] columns =
+ new Column[] {
+ Column.of("id", Types.IntegerType.get(), "id"),
+ Column.of("payload", Types.NullType.get(), "unknown col")
+ };
+ Map<String, String> properties = Maps.newHashMap();
+ // unknown is a V3 type, so the table must be created at format-version 3.
+ properties.put(IcebergTablePropertiesMetadata.FORMAT_VERSION, "3");
+
+ TableCatalog tableCatalog = catalog.asTableCatalog();
+ Table created =
+ tableCatalog.createTable(
+ ident,
+ columns,
+ "unknown write",
+ properties,
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ new SortOrder[0]);
+ Assertions.assertInstanceOf(Types.NullType.class,
created.columns()[1].dataType());
+
+ // Load it back through the native metadata API to confirm the round-trip
against the backend.
+ Table loaded = tableCatalog.loadTable(ident);
+ Assertions.assertInstanceOf(Types.NullType.class,
loaded.columns()[1].dataType());
+ Assertions.assertEquals(
+ "3",
loaded.properties().get(IcebergTablePropertiesMetadata.FORMAT_VERSION));
+
+ // Cross-surface check (native write -> IRC read): the native write
persisted a real Iceberg
+ // `unknown` column, so reading the same table through the Iceberg REST
(IRC) API returns
+ // unknown. This is the mirror of testV3TypeConversionViaIcebergClient
(IRC write -> native
+ // read).
+ org.apache.iceberg.Table icebergTable =
+
icebergCatalog.loadTable(IcebergCatalogWrapperHelper.buildIcebergTableIdentifier(ident));
+ Assertions.assertEquals(
+ org.apache.iceberg.types.Types.UnknownType.get(),
+ icebergTable.schema().findField("payload").type());
+ }
+
@Test
void testCreateGeometryColumnWriteRoundTrip() {
// Write path: create a geometry column with a non-default CRS *through
the Gravitino relational
@@ -996,6 +1044,13 @@ public abstract class CatalogIcebergBaseIT extends BaseIT
{
Assertions.assertInstanceOf(Types.VariantType.class, loaded.dataType());
}
+ private void assertV3LoadsAsNull(String tableName) {
+ NameIdentifier ident =
+ createV3Table(tableName,
org.apache.iceberg.types.Types.UnknownType.get());
+ Column loaded = catalog.asTableCatalog().loadTable(ident).columns()[0];
+ Assertions.assertInstanceOf(Types.NullType.class, loaded.dataType());
+ }
+
private void assertV3LoadsAsGeometry(
String tableName, org.apache.iceberg.types.Types.GeometryType
icebergType) {
NameIdentifier ident = createV3Table(tableName, icebergType);
diff --git a/docs/lakehouse-iceberg-catalog.md
b/docs/lakehouse-iceberg-catalog.md
index 3be17a3104..e8d3ba94d5 100644
--- a/docs/lakehouse-iceberg-catalog.md
+++ b/docs/lakehouse-iceberg-catalog.md
@@ -437,9 +437,17 @@ If you doesn't specify distribution expressions, the table
distribution will be
| `Binary` | `Binary` |
| `UUID` | `UUID` |
| `Variant` | `Variant` |
+| `Null` | `Unknown` |
| `Geometry` | `Geometry` |
| `Geography` | `Geography` |
+:::note
+Gravitino `Null` maps to Apache Iceberg's V3 `unknown` type — a null-only
placeholder for a column
+whose type is not yet known. It requires table `format-version` 3, must be an
optional (nullable)
+column, and can be promoted to a concrete type via schema evolution. This is
the recommended way to
+represent an Iceberg `unknown` column in Gravitino.
+:::
+
:::info
Apache Iceberg doesn't support Gravitino `Varchar` `Fixedchar` `Byte` `Short`
`Union` type.
Meanwhile, the data types other than listed above are mapped to Gravitino
**[External
Type](./manage-relational-metadata-using-gravitino.md#external-type)** that
represents an unresolvable data type since 0.6.0-incubating.
diff --git a/docs/manage-relational-metadata-using-gravitino.md
b/docs/manage-relational-metadata-using-gravitino.md
index ebdace5c19..f8e99974bd 100644
--- a/docs/manage-relational-metadata-using-gravitino.md
+++ b/docs/manage-relational-metadata-using-gravitino.md
@@ -931,11 +931,24 @@ The following types that Gravitino supports:
| Union | `Types.UnionType.of([type1, type2, ...])`
| `{"type": "union", "types": [type JSON, ...]}`
| Union type, indicates a union of types
|
| UUID | `Types.UUIDType.get()`
| `uuid`
| UUID type, indicates a universally unique identifier
|
| Variant | `Types.VariantType.get()`
| `variant`
| Variant type, indicates semi-structured data whose shape is not fixed by the
schema
|
+| Null | `Types.NullType.get()`
| `null`
| Null type, indicates a column that holds only null values and whose concrete
type is not yet known; it may be assigned a specific type later via schema
evolution |
| Geometry | `Types.GeometryType.of(crs)`
| `geometry` or `geometry(<crs>)`
| Geometry type, indicates a geospatial shape (WKB-encoded) on a planar
coordinate reference system (CRS), default `OGC:CRS84`
|
| Geography | `Types.GeographyType.of(crs, algorithm)`
| `geography` or `geography(<crs>,<algorithm>)`
| Geography type, indicates a geospatial shape (WKB-encoded) on a spheroidal
CRS (default `OGC:CRS84`) with an edge-interpolation algorithm (default
`spherical`) |
The related java doc is
[here](pathname:///docs/2.0.0-SNAPSHOT/api/java/org/apache/gravitino/rel/types/Type.html).
+##### Null type
+
+The null type represents a column that holds only null values and whose
concrete type is not yet
+known — it carries no usable data of its own, and the intent is that a more
specific type is
+assigned later through schema evolution. This mirrors the "null"/"void" type
in engines such as
+Spark, Flink, and Arrow, and maps to Apache Iceberg's V3 `unknown` type.
+
+Support is connector-specific and still expanding: it is mapped natively by
the Iceberg, Spark,
+Flink, and Lance connectors, while connectors without an equivalent concept
currently reject it.
+Because a null column holds no usable data, engines should not populate it; it
is meant to be
+promoted to a concrete type before data is written.
+
##### External type
External type is a special type of column type, when you need to use a data
type that is not in the Gravitino type
diff --git a/docs/open-api/datatype.yaml b/docs/open-api/datatype.yaml
index 0cc75b5e19..d924754bbc 100644
--- a/docs/open-api/datatype.yaml
+++ b/docs/open-api/datatype.yaml
@@ -56,6 +56,7 @@ components:
- "fixed(16)"
- "binary"
- "variant"
+ - "null"
- "geometry"
- "geometry(srid:3857)"
- "geography"
@@ -221,5 +222,5 @@ components:
description: The string representation of this type in the catalog
example: {
"type": "external",
- "externalType": "user-defined"
- }
\ No newline at end of file
+ "catalogString": "user-defined"
+ }
