This is an automated email from the ASF dual-hosted git repository.
mchades pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 99941e2d38 [#11949] feat(iceberg): native variant type support (#11932)
99941e2d38 is described below
commit 99941e2d38f8978be66efb09015b5f025cfcb5ce
Author: Nevin Zheng <[email protected]>
AuthorDate: Wed Jul 8 21:56:26 2026 -0700
[#11949] feat(iceberg): native variant type support (#11932)
### What changes were proposed in this pull request?
Add a first-class `variant` type to Gravitino's unified type model and
wire it through the Iceberg catalog and the Python client. At the schema
level Iceberg V3's `variant` is an opaque, parameterless leaf, so
`VariantType` is a singleton `PrimitiveType` like `UUIDType`.
- `api`: `Type.Name.VARIANT` + `Types.VariantType`.
- `common`: `JsonUtils` serializes/parses it as the token `"variant"`.
- `catalog-lakehouse-iceberg`: map Iceberg `VariantType` ⇄ Gravitino
`VariantType`.
- Python client: mirror `Name.VARIANT`, `Types.VariantType`, and the
serde registry, so a Python client loads a variant column as
`VariantType` rather than `UnparsedType`.
- Docs: unified type reference, Iceberg type-mapping table, OpenAPI
examples.
Fixes #11949
**Follow-ups:** this is the first of a short series, split for
reviewability. Later PRs add native `variant` support for the other
engines that have Open Variant (Paimon, Doris), then test coverage
locking in reject behavior for the engines that don't. They'll go up one
at a time as each merges.
### Why are the changes needed?
Loading an Iceberg V3 table with a `variant` column through the native
metadata API failed with `UnsupportedOperationException: Unsupported
type: variant` (#11927). Modeling variant natively rather than as an
opaque `ExternalType` string gives it a stable identity so each
connector maps it deliberately. Discussed in #11929; supersedes the
`ExternalType` stopgap (#11928).
### Does this PR introduce any user-facing change?
Yes — Iceberg V3 tables with a `variant` column now load through the
native API as a `variant` type. Other connectors are unchanged in this
PR.
### How was this patch tested?
- Unit: `TestTypes` (type contract), `TestJsonUtils` (JSON round-trip),
`TestConvertUtil` (Iceberg converter, both directions).
- Python unit: variant type contract + serde round-trip.
- Docker IT: `CatalogIcebergBaseIT` — create a format-version-3 table
via the raw Iceberg client, load it through the native API, and assert
variant resolves to the native type.
---
.../java/org/apache/gravitino/rel/types/Type.java | 5 ++
.../java/org/apache/gravitino/rel/types/Types.java | 24 ++++++
.../java/org/apache/gravitino/rel/TestTypes.java | 5 ++
.../iceberg/converter/FromIcebergType.java | 5 ++
.../lakehouse/iceberg/converter/ToIcebergType.java | 2 +
.../iceberg/converter/TestConvertUtil.java | 11 +++
.../integration/test/CatalogIcebergBaseIT.java | 93 ++++++++++++++++++++++
.../client-python/gravitino/api/rel/types/type.py | 4 +
.../client-python/gravitino/api/rel/types/types.py | 22 +++++
clients/client-python/gravitino/utils/serdes.py | 1 +
.../tests/unittests/api/rel/test_types.py | 6 ++
.../java/org/apache/gravitino/json/JsonUtils.java | 2 +
.../org/apache/gravitino/json/TestJsonUtils.java | 7 ++
docs/lakehouse-iceberg-catalog.md | 1 +
docs/manage-relational-metadata-using-gravitino.md | 1 +
docs/open-api/datatype.yaml | 1 +
16 files changed, 190 insertions(+)
diff --git a/api/src/main/java/org/apache/gravitino/rel/types/Type.java
b/api/src/main/java/org/apache/gravitino/rel/types/Type.java
index 4fb4d740c2..223762259c 100644
--- a/api/src/main/java/org/apache/gravitino/rel/types/Type.java
+++ b/api/src/main/java/org/apache/gravitino/rel/types/Type.java
@@ -73,6 +73,11 @@ public interface Type {
FIXED,
/** The binary type with variable length. The length is specified in the
type itself. */
BINARY,
+ /**
+ * The variant type. A variant holds semi-structured data whose shape is
not fixed by the
+ * schema.
+ */
+ VARIANT,
/**
* The struct type. A struct type is a complex type that contains a set of
named fields, each
* with a type, and optionally a comment.
diff --git a/api/src/main/java/org/apache/gravitino/rel/types/Types.java
b/api/src/main/java/org/apache/gravitino/rel/types/Types.java
index be785124ff..11484db648 100644
--- a/api/src/main/java/org/apache/gravitino/rel/types/Types.java
+++ b/api/src/main/java/org/apache/gravitino/rel/types/Types.java
@@ -655,6 +655,30 @@ public class Types {
}
}
+ /** The variant type in Gravitino, holding semi-structured data not
described by the schema. */
+ public static class VariantType extends Type.PrimitiveType {
+ private static final VariantType INSTANCE = new VariantType();
+
+ /**
+ * @return The singleton instance of {@link VariantType}.
+ */
+ public static VariantType get() {
+ return INSTANCE;
+ }
+
+ private VariantType() {}
+
+ @Override
+ public Name name() {
+ return Name.VARIANT;
+ }
+
+ @Override
+ public String simpleString() {
+ return "variant";
+ }
+ }
+
/**
* Fixed-length byte array type, if you want to use variable-length byte
array, use {@link
* BinaryType} instead.
diff --git a/api/src/test/java/org/apache/gravitino/rel/TestTypes.java
b/api/src/test/java/org/apache/gravitino/rel/TestTypes.java
index 75da38db6e..8b58751721 100644
--- a/api/src/test/java/org/apache/gravitino/rel/TestTypes.java
+++ b/api/src/test/java/org/apache/gravitino/rel/TestTypes.java
@@ -154,6 +154,11 @@ public class TestTypes {
Assertions.assertSame(uuidType, Types.UUIDType.get());
Assertions.assertEquals("uuid", uuidType.simpleString());
+ Types.VariantType variantType = Types.VariantType.get();
+ Assertions.assertEquals(Type.Name.VARIANT, variantType.name());
+ Assertions.assertSame(variantType, Types.VariantType.get());
+ Assertions.assertEquals("variant", variantType.simpleString());
+
Types.FixedType fixedType = Types.FixedType.of(10);
Assertions.assertEquals(Type.Name.FIXED, fixedType.name());
Assertions.assertEquals(10, fixedType.length());
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/FromIcebergType.java
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/FromIcebergType.java
index e47bf34670..02b8be1dbd 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/FromIcebergType.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/FromIcebergType.java
@@ -76,6 +76,11 @@ public class FromIcebergType extends
TypeUtil.SchemaVisitor<Type> {
keyResult, valueResult, map.isValueOptional());
}
+ @Override
+ public Type variant(Types.VariantType variant) {
+ return org.apache.gravitino.rel.types.Types.VariantType.get();
+ }
+
@Override
public Type primitive(org.apache.iceberg.types.Type.PrimitiveType primitive)
{
switch (primitive.typeId()) {
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergType.java
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergType.java
index 58b88c626d..b41e29cf12 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergType.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ToIcebergType.java
@@ -162,6 +162,8 @@ public class ToIcebergType extends
ToIcebergTypeVisitor<Type> {
return Types.BinaryType.get();
} else if (primitive instanceof
org.apache.gravitino.rel.types.Types.UUIDType) {
return Types.UUIDType.get();
+ } else if (primitive instanceof
org.apache.gravitino.rel.types.Types.VariantType) {
+ return Types.VariantType.get();
}
throw new UnsupportedOperationException("Not a supported type: " +
primitive.toString());
}
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/TestConvertUtil.java
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/TestConvertUtil.java
index f9099666d4..659f180a2c 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/TestConvertUtil.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/TestConvertUtil.java
@@ -43,6 +43,17 @@ import org.junit.jupiter.api.Test;
/** Test class for {@link ConvertUtil}. */
public class TestConvertUtil extends TestBaseConvert {
+
+ @Test
+ public void testVariantType() {
+ Assertions.assertTrue(
+ CONVERTER.toGravitino(Types.VariantType.get())
+ instanceof org.apache.gravitino.rel.types.Types.VariantType);
+ Assertions.assertTrue(
+
CONVERTER.fromGravitino(org.apache.gravitino.rel.types.Types.VariantType.get())
+ instanceof Types.VariantType);
+ }
+
@Test
public void testToIcebergSchema() {
Column[] columns = createColumns("col_1", "col_2", "col_3", "col_4");
diff --git
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergBaseIT.java
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergBaseIT.java
index f4748d82ad..f7b6770ca4 100644
---
a/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergBaseIT.java
+++
b/catalogs/catalog-lakehouse-iceberg/src/test/java/org/apache/gravitino/catalog/lakehouse/iceberg/integration/test/CatalogIcebergBaseIT.java
@@ -40,6 +40,7 @@ import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
import org.apache.commons.lang3.ArrayUtils;
+import org.apache.commons.lang3.exception.ExceptionUtils;
import org.apache.gravitino.Catalog;
import org.apache.gravitino.NameIdentifier;
import org.apache.gravitino.Namespace;
@@ -97,6 +98,7 @@ import org.apache.spark.sql.SparkSession;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Assumptions;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
@@ -629,6 +631,97 @@ public abstract class CatalogIcebergBaseIT extends BaseIT {
Assertions.assertEquals("col_2_comment",
icebergSchema.columns().get(1).doc());
}
+ @Test
+ void testV3TypeConversionViaIcebergClient() {
+ // REST-backend only. The Hive metastore in the current CI Hive image
cannot store Iceberg V3
+ // column types, so a V3 table cannot be created through the Hive backend
at all. That behavior
+ // is captured in testV3TypesRejectedByHiveMetastore. Once Hive gains V3
support (HIVE-29192 /
+ // HIVE-29287) and the CI Hive image is bumped, this test and the Hive
test can be merged.
+ Assumptions.assumeTrue(
+ "rest".equalsIgnoreCase(TYPE),
+ "V3 types require a backend that does not validate against the Hive
metastore");
+
+ // These types cannot be created through the native interface, so create
them the way an engine
+ // like Spark 4 would, directly through the Iceberg catalog, then load
through the native
+ // metadata interface.
+ //
+ // variant has native support and loads as VariantType. The other V3
net-new types are not
+ // modeled in Gravitino's unified type system yet and load as
ExternalType; native support for
+ // them is pending and tracked in apache/gravitino#11929.
+ assertV3LoadsAsVariant("v3_variant");
+
+ // TODO(apache/gravitino#11929): expect native types once these gain
unified-model support.
+ assertV3LoadsAsExternal(
+ "v3_timestamp_ns",
+ org.apache.iceberg.types.Types.TimestampNanoType.withoutZone(),
+ "TIMESTAMP_NANO");
+ assertV3LoadsAsExternal(
+ "v3_timestamptz_ns",
+ org.apache.iceberg.types.Types.TimestampNanoType.withZone(),
+ "TIMESTAMP_NANO");
+ assertV3LoadsAsExternal(
+ "v3_geometry", org.apache.iceberg.types.Types.GeometryType.crs84(),
"GEOMETRY");
+ assertV3LoadsAsExternal(
+ "v3_geography", org.apache.iceberg.types.Types.GeographyType.crs84(),
"GEOGRAPHY");
+ assertV3LoadsAsExternal(
+ "v3_unknown", org.apache.iceberg.types.Types.UnknownType.get(),
"UNKNOWN");
+ }
+
+ @Test
+ void testV3TypesRejectedByHiveMetastore() {
+ // Hive-backend only. The Hive metastore in the current CI Hive image has
no mapping for Iceberg
+ // V3 column types, so creating a V3 table through the Hive backend fails
at metastore
+ // registration ("Invalid column type"). This captures that current
behavior so it is not
+ // silently lost. When Hive gains V3 support (HIVE-29192 / HIVE-29287) and
the CI Hive image is
+ // bumped, this create will start to succeed, this test will fail, and it
can be merged with
+ // testV3TypeConversionViaIcebergClient. See apache/gravitino#11929.
+ Assumptions.assumeTrue(
+ "hive".equalsIgnoreCase(TYPE), "Only the Hive metastore backend
rejects V3 column types");
+
+ Exception exception =
+ Assertions.assertThrows(
+ Exception.class,
+ () ->
+ createV3Table("v3_variant_hive",
org.apache.iceberg.types.Types.VariantType.get()));
+ // Assert the failure is specifically the Hive metastore rejecting the V3
column type, not an
+ // unrelated wiring or connection error, so this test cannot pass for the
wrong reason.
+ Assertions.assertTrue(
+ ExceptionUtils.getStackTrace(exception).contains("Invalid column
type"),
+ "Expected a Hive metastore 'Invalid column type' rejection, but got: "
+ exception);
+ }
+
+ private NameIdentifier createV3Table(
+ String tableName, org.apache.iceberg.types.Type icebergType) {
+ NameIdentifier ident = NameIdentifier.of(schemaName, tableName);
+ org.apache.iceberg.Schema schema =
+ new org.apache.iceberg.Schema(
+ org.apache.iceberg.types.Types.NestedField.optional(1, "c",
icebergType));
+ Map<String, String> props = Maps.newHashMap();
+ props.put("format-version", "3");
+ icebergCatalog.createTable(
+ IcebergCatalogWrapperHelper.buildIcebergTableIdentifier(ident),
+ schema,
+ org.apache.iceberg.PartitionSpec.unpartitioned(),
+ props);
+ return ident;
+ }
+
+ private void assertV3LoadsAsVariant(String tableName) {
+ NameIdentifier ident =
+ createV3Table(tableName,
org.apache.iceberg.types.Types.VariantType.get());
+ Column loaded = catalog.asTableCatalog().loadTable(ident).columns()[0];
+ Assertions.assertInstanceOf(Types.VariantType.class, loaded.dataType());
+ }
+
+ private void assertV3LoadsAsExternal(
+ String tableName, org.apache.iceberg.types.Type icebergType, String
expectedCatalogString) {
+ NameIdentifier ident = createV3Table(tableName, icebergType);
+ Column loaded = catalog.asTableCatalog().loadTable(ident).columns()[0];
+ Assertions.assertInstanceOf(Types.ExternalType.class, loaded.dataType());
+ Assertions.assertEquals(
+ expectedCatalogString, ((Types.ExternalType)
loaded.dataType()).catalogString());
+ }
+
@Test
void testListAndDropIcebergTable() {
Column[] columns = createColumns();
diff --git a/clients/client-python/gravitino/api/rel/types/type.py
b/clients/client-python/gravitino/api/rel/types/type.py
index bbc467a05e..7cf9f921d8 100644
--- a/clients/client-python/gravitino/api/rel/types/type.py
+++ b/clients/client-python/gravitino/api/rel/types/type.py
@@ -81,6 +81,10 @@ class Name(Enum):
BINARY = "BINARY"
""" The binary type with variable length. The length is specified in the
type itself. """
+ VARIANT = "VARIANT"
+ """ The variant type. A variant holds semi-structured data whose shape is
not fixed by the
+ schema. """
+
STRUCT = "STRUCT"
"""
The struct type.
diff --git a/clients/client-python/gravitino/api/rel/types/types.py
b/clients/client-python/gravitino/api/rel/types/types.py
index f182f086bf..622c4ac8cc 100644
--- a/clients/client-python/gravitino/api/rel/types/types.py
+++ b/clients/client-python/gravitino/api/rel/types/types.py
@@ -706,6 +706,28 @@ class Types:
def simple_string(self) -> str:
return "binary"
+ class VariantType(PrimitiveType):
+ """The variant type in Gravitino, holding semi-structured data not
described by the
+ schema."""
+
+ _instance: Types.VariantType = None
+
+ def __new__(cls):
+ if cls._instance is None:
+ cls._instance = super(Types.VariantType, cls).__new__(cls)
+ cls._instance.__init__()
+ return cls._instance
+
+ @classmethod
+ def get(cls) -> Types.VariantType:
+ return cls()
+
+ def name(self) -> Name:
+ return Name.VARIANT
+
+ def simple_string(self) -> str:
+ return "variant"
+
class StructType(ComplexType):
"""The struct type in Gravitino."""
diff --git a/clients/client-python/gravitino/utils/serdes.py
b/clients/client-python/gravitino/utils/serdes.py
index ddee51ac84..fa5fbe3c80 100644
--- a/clients/client-python/gravitino/utils/serdes.py
+++ b/clients/client-python/gravitino/utils/serdes.py
@@ -116,6 +116,7 @@ class SerdesUtilsBase:
Types.IntervalDayType.get(),
Types.StringType.get(),
Types.UUIDType.get(),
+ Types.VariantType.get(),
)
}
)
diff --git a/clients/client-python/tests/unittests/api/rel/test_types.py
b/clients/client-python/tests/unittests/api/rel/test_types.py
index 940b11f99d..4cf0d41cc9 100644
--- a/clients/client-python/tests/unittests/api/rel/test_types.py
+++ b/clients/client-python/tests/unittests/api/rel/test_types.py
@@ -182,6 +182,12 @@ class TestTypes(unittest.TestCase):
self.assertEqual(instance.name(), Name.UUID)
self.assertEqual(instance.simple_string(), "uuid")
+ def test_variant_type(self):
+ instance: Types.VariantType = Types.VariantType.get()
+ self.assertEqual(instance.name(), Name.VARIANT)
+ self.assertEqual(instance.simple_string(), "variant")
+ self.assertIs(instance, Types.VariantType.get())
+
def test_fixed_type(self):
instance: Types.FixedType = Types.FixedType.of(5)
self.assertEqual(instance.name(), Name.FIXED)
diff --git a/common/src/main/java/org/apache/gravitino/json/JsonUtils.java
b/common/src/main/java/org/apache/gravitino/json/JsonUtils.java
index 2b664acc36..5c87d28566 100644
--- a/common/src/main/java/org/apache/gravitino/json/JsonUtils.java
+++ b/common/src/main/java/org/apache/gravitino/json/JsonUtils.java
@@ -170,6 +170,7 @@ public class JsonUtils {
Types.TimestampType.withoutTimeZone(),
Types.StringType.get(),
Types.UUIDType.get(),
+ Types.VariantType.get(),
Types.BinaryType.get(),
Types.IntervalYearType.get(),
Types.IntervalDayType.get()),
@@ -680,6 +681,7 @@ public class JsonUtils {
case INTERVAL_DAY:
case INTERVAL_YEAR:
case UUID:
+ case VARIANT:
case FIXED:
case BINARY:
case NULL:
diff --git a/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
b/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
index c99af454e3..da92570e1d 100644
--- a/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
+++ b/common/src/test/java/org/apache/gravitino/json/TestJsonUtils.java
@@ -119,6 +119,13 @@ public class TestJsonUtils {
expected = "\"decimal(10,2)\"";
Assertions.assertEquals(objectMapper.readTree(expected),
objectMapper.readTree(jsonValue));
+ type = Types.VariantType.get();
+ jsonValue = JsonUtils.objectMapper().writeValueAsString(type);
+ expected = "\"variant\"";
+ Assertions.assertEquals(objectMapper.readTree(expected),
objectMapper.readTree(jsonValue));
+ Assertions.assertEquals(
+ Types.VariantType.get(), JsonUtils.objectMapper().readValue(jsonValue,
Type.class));
+
type =
Types.StructType.of(
Types.StructType.Field.nullableField("name",
Types.StringType.get(), "name field"),
diff --git a/docs/lakehouse-iceberg-catalog.md
b/docs/lakehouse-iceberg-catalog.md
index 0badff0441..5f11e68744 100644
--- a/docs/lakehouse-iceberg-catalog.md
+++ b/docs/lakehouse-iceberg-catalog.md
@@ -434,6 +434,7 @@ If you doesn't specify distribution expressions, the table
distribution will be
| `Fixed` | `Fixed` |
| `Binary` | `Binary` |
| `UUID` | `UUID` |
+| `Variant` | `Variant` |
:::info
Apache Iceberg doesn't support Gravitino `Varchar` `Fixedchar` `Byte` `Short`
`Union` type.
diff --git a/docs/manage-relational-metadata-using-gravitino.md
b/docs/manage-relational-metadata-using-gravitino.md
index 6bbbb50e94..cfda9dc231 100644
--- a/docs/manage-relational-metadata-using-gravitino.md
+++ b/docs/manage-relational-metadata-using-gravitino.md
@@ -930,6 +930,7 @@ The following types that Gravitino supports:
| Struct |
`Types.StructType.of([Types.StructType.Field.of(name, type, nullable)])` |
`{"type": "struct", "fields": [JSON StructField, {"name": string, "type": type
JSON, "nullable": JSON Boolean, "comment": string}]}` | Struct type, indicate a
struct of fields
|
| Union | `Types.UnionType.of([type1, type2, ...])`
| `{"type": "union", "types": [type JSON, ...]}`
| Union type, indicates a union of types
|
| UUID | `Types.UUIDType.get()`
| `uuid`
| UUID type, indicates a universally unique identifier
|
+| Variant | `Types.VariantType.get()`
| `variant`
| Variant type, indicates semi-structured data whose shape is not fixed by the
schema
|
The related java doc is
[here](pathname:///docs/2.0.0-SNAPSHOT/api/java/org/apache/gravitino/rel/types/Type.html).
diff --git a/docs/open-api/datatype.yaml b/docs/open-api/datatype.yaml
index d5ad0cc2b5..a244aee188 100644
--- a/docs/open-api/datatype.yaml
+++ b/docs/open-api/datatype.yaml
@@ -55,6 +55,7 @@ components:
- "uuid"
- "fixed(16)"
- "binary"
+ - "variant"
UnionType:
type: object
