Hello Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24521
to look at the new patch set (#3).
Change subject: IMPALA-15052: Add read support for unshredded VARIANT values
......................................................................
IMPALA-15052: Add read support for unshredded VARIANT values
Builds on the backend VARIANT decoding from IMPALA-15067 to enable
end-to-end reading of unshredded VARIANT columns from Iceberg tables
stored as Parquet. VARIANT is read-only: it can be scanned and rendered
as JSON, but it cannot be created or written, and it can only appear in
queries as a pass-through column reference.
Type system:
- TYPE_VARIANT added across thrift (TPrimitiveType/TTypeNodeType), the
BE (PrimitiveType/ColumnType, 24-byte slot = two StringValues) and the
FE (VariantType extending Type with struct-like metadata/value fields)
- An unshredded VARIANT is modeled as a two-child group: "metadata" and
"value", both BINARY.
Read path (Parquet):
- VariantColumnReader (extends StructColumnReader) reads the variant
group's metadata + value children. The physical layout matches a
two-field struct, so ParquetSchemaResolver resolves VARIANT like
STRUCT.
- A malformed variant group whose Parquet child count does not match the
catalog (i.e. not exactly metadata + value) fails with a runtime
error.
- The reader is excluded from late materialization (it inherits the
struct SkipRows() restriction) so a filtered scan never calls
SkipRows() on it.
- FE descriptors: the Analyzer builds a children tuple with
metadata/value slots for each VARIANT column.
Result display:
- VARIANT columns are serialized to their JSON representation in query
output (hs2-util, query-result-set), via the backend VariantSlotToJson
helper.
- A VARIANT value that fails to decode (corruption) is surfaced as
SQL NULL with a bounded warning log.
DDL and analysis (VARIANT is read-only):
- The VARIANT keyword is recognized by the parser but rejected as a
user-specified column type or CAST target (CREATE/ALTER TABLE, CAST).
- VARIANT raises an AnalysisException in ORDER BY, GROUP BY,
SELECT DISTINCT, UNION/INTERSECT/EXCEPT, CASE/DECODE, IS NULL,
analytic PARTITION BY/ORDER BY, aggregate DISTINCT, INSERT, and
COMPUTE STATS.
- UNION ALL of a variant column is supported.
- Only a top-level VARIANT column can be queried. A VARIANT nested
inside another complex type (ARRAY<VARIANT>, MAP<..,VARIANT>,
STRUCT<..VARIANT..>, and deeper combinations) is rejected. Such a
table still loads; only queries that reference the nested VARIANT fail
Iceberg / HMS:
- IcebergSchemaConverter maps the Iceberg VARIANT type id to
TYPE_VARIANT.
- HMS has no 'variant' column type, so the new Type.toMetastoreString()
stores VARIANT as its underlying struct<metadata:binary,value:binary>
(matching Iceberg's Hive catalog and Trino). Impala still reads the
real VARIANT type from the Iceberg metadata; the HMS FieldSchema is
only used for Hive interop.
Codegen:
- Codegen is disabled for queries that scan a VARIANT column (currently
query-wide; see TODO in HdfsScanNode), and
Tuple::CodegenMaterializeExprs falls back to the interpreted path for
VARIANT slots. The interpreted infrastructure treats VARIANT like
STRUCT to avoid crashes.
Forward compatibility:
- No user-facing variant_get()/variant_to_json() builtin is registered
yet. These will be added once VARIANT is a first-class expression type
(a VariantVal ABI that lets functions accept and return VARIANT).
Testing:
- New e2e test (iceberg-v3-variant.test) covering JSON rendering of
primitives/strings/arrays/objects, filtered scans, TOP-N, UNION ALL,
and the negative analysis cases above, against a Trino-written Iceberg
V3 table (trino_variant).
- The same test also covers the nested-VARIANT rejections against a
second Trino-written table (trino_nested_variant) that carries VARIANT
nested in ARRAY/MAP/STRUCT (and ARRAY<STRUCT<VARIANT>>), plus a
top-level VARIANT column to confirm that case still works. It also
checks DESCRIBE and DESCRIBE FORMATTED rendering of the VARIANT and
nested-VARIANT columns (describing such a table is always allowed).
Change-Id: Ie2f8a7c9b1d4e5f6a0c3b8d7e9f1a2b4c6d8e0f1
Assisted-by: Claude Opus 4.6 <[email protected]>
Assisted-by: Claude Opus 4.8 (1M context) <[email protected]>
---
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/exec/parquet/parquet-struct-column-reader.h
A be/src/exec/parquet/parquet-variant-column-reader.h
M be/src/exprs/anyval-util.cc
M be/src/exprs/scalar-expr-evaluator.cc
M be/src/exprs/scalar-expr.cc
M be/src/exprs/scalar-expr.inline.h
M be/src/exprs/slot-ref.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/raw-value.cc
M be/src/runtime/raw-value.h
M be/src/runtime/tuple.cc
M be/src/runtime/types.cc
M be/src/runtime/types.h
M be/src/service/hs2-util.cc
M be/src/service/query-result-set.cc
M be/src/util/variant-util.cc
M be/src/util/variant-util.h
M common/thrift/Types.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/CaseExpr.java
M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/Path.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/TypeDef.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
A fe/src/main/java/org/apache/impala/catalog/VariantType.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/main/jflex/sql-scanner.flex
A
testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/data/20260703_115517_00018_wa2kj-fc2bf4c0-5cb6-4bd2-abb2-2eac147adff9.parquet
A
testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/20260703_115517_00018_wa2kj-6375d4da-66f3-4adf-b0d6-fb7964f5b953.stats
A
testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/f8d5c650-108d-46aa-ae9f-e9d94e10adf4-m0.avro
A
testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/snap-5173553188971266738-1-f8d5c650-108d-46aa-ae9f-e9d94e10adf4.avro
A
testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/snap-5208362247809940782-1-10d5391c-e63e-4913-b898-ad7ef55cf306.avro
A
testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/v1.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/v2.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/trino_nested_variant/metadata/version-hint.text
A
testdata/data/iceberg_test/iceberg_v3/trino_variant/data/20260629_095655_00023_wa2kj-0f39c7a5-6cc6-4be7-abc0-a45a62dacec5.parquet
A
testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/20260629_095655_00023_wa2kj-bcc1a100-d380-42cf-bccf-2e32badf7ac5.stats
A
testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/a874fc23-2c2c-42b5-9fc0-f72919a7449d-m0.avro
A
testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/snap-6097489456815673056-1-a874fc23-2c2c-42b5-9fc0-f72919a7449d.avro
A
testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/snap-8332248857076198293-1-03e34f74-1a29-4e53-ab2c-6f2c507d534d.avro
A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/v1.metadata.json
A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/v2.metadata.json
A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/version-hint.text
A testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-variant.test
M tests/query_test/test_iceberg.py
M tests/util/iceberg_metadata_util.py
67 files changed, 1,040 insertions(+), 65 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/24521/3
--
To view, visit http://gerrit.cloudera.org:8080/24521
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie2f8a7c9b1d4e5f6a0c3b8d7e9f1a2b4c6d8e0f1
Gerrit-Change-Number: 24521
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
