Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24521
Change subject: IMPALA-15052: Add read support for unshredded VARIANT values ...................................................................... IMPALA-15052: Add read support for unshredded VARIANT values Builds on the backend VARIANT decoding from IMPALA-15067 to enable end-to-end reading of unshredded VARIANT columns from Iceberg tables stored as Parquet. VARIANT is read-only: it can be scanned and rendered as JSON, but it cannot be created or written, and it can only appear in queries as a pass-through column reference. Type system: - TYPE_VARIANT added across thrift (TPrimitiveType/TTypeNodeType), the BE (PrimitiveType/ColumnType, 24-byte slot = two StringValues) and the FE (VariantType extending Type with struct-like metadata/value fields) - An unshredded VARIANT is modeled as a two-child group: "metadata" and "value", both BINARY. Read path (Parquet): - VariantColumnReader (extends StructColumnReader) reads the variant group's metadata + value children. The physical layout matches a two-field struct, so ParquetSchemaResolver resolves VARIANT like STRUCT. - A malformed variant group whose Parquet child count does not match the catalog (i.e. not exactly metadata + value) fails with a runtime error. - The reader is excluded from late materialization (it inherits the struct SkipRows() restriction) so a filtered scan never calls SkipRows() on it. - FE descriptors: the Analyzer builds a children tuple with metadata/value slots for each VARIANT column. Result display: - VARIANT columns are serialized to their JSON representation in query output (hs2-util, query-result-set). The variant_to_json(BINARY, BINARY) -> STRING builtin is also registered. - A VARIANT value that fails to decode (corruption) is surfaced as SQL NULL with a bounded warning log. DDL and analysis (VARIANT is read-only): - The VARIANT keyword is recognized by the parser but rejected as a user-specified column type or CAST target (CREATE/ALTER TABLE, CAST). - VARIANT raises an AnalysisException in ORDER BY, GROUP BY, SELECT DISTINCT, UNION/INTERSECT/EXCEPT, CASE/DECODE, IS NULL, analytic PARTITION BY/ORDER BY, aggregate DISTINCT, INSERT, and COMPUTE STATS. - UNION ALL of a variant column is supported. Iceberg / HMS: - IcebergSchemaConverter maps the Iceberg VARIANT type id to TYPE_VARIANT. - HMS has no 'variant' column type, so the new Type.toMetastoreString() stores VARIANT as its underlying struct<metadata:binary,value:binary> (matching Iceberg's Hive catalog and Trino). Impala still reads the real VARIANT type from the Iceberg metadata; the HMS FieldSchema is only used for Hive interop. Codegen: - Codegen is disabled for queries that scan a VARIANT column (currently query-wide; see TODO in HdfsScanNode), and Tuple::CodegenMaterializeExprs falls back to the interpreted path for VARIANT slots. The interpreted infrastructure treats VARIANT like STRUCT to avoid crashes. Testing: - New e2e test (iceberg-v3-variant.test) covering JSON rendering of primitives/strings/arrays/objects, filtered scans, TOP-N, UNION ALL, and the negative analysis cases above, against a Trino-written Iceberg V3 table. Change-Id: Ie2f8a7c9b1d4e5f6a0c3b8d7e9f1a2b4c6d8e0f1 Assisted-by: Claude Opus 4.6 <[email protected]> Assisted-by: Claude Opus 4.8 (1M context) <[email protected]> --- M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-struct-column-reader.h A be/src/exec/parquet/parquet-variant-column-reader.h M be/src/exprs/anyval-util.cc M be/src/exprs/scalar-expr-evaluator.cc M be/src/exprs/scalar-expr.cc M be/src/exprs/scalar-expr.inline.h M be/src/exprs/slot-ref.cc M be/src/runtime/descriptors.cc M be/src/runtime/raw-value.cc M be/src/runtime/raw-value.h M be/src/runtime/tuple.cc M be/src/runtime/types.cc M be/src/runtime/types.h M be/src/service/hs2-util.cc M be/src/service/query-result-set.cc M be/src/util/variant-util.cc M be/src/util/variant-util.h M common/function-registry/impala_functions.py M common/thrift/Types.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CaseExpr.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/TypeDef.java M fe/src/main/java/org/apache/impala/catalog/Type.java A fe/src/main/java/org/apache/impala/catalog/VariantType.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/main/jflex/sql-scanner.flex A testdata/data/iceberg_test/iceberg_v3/trino_variant/data/20260629_095655_00023_wa2kj-0f39c7a5-6cc6-4be7-abc0-a45a62dacec5.parquet A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/20260629_095655_00023_wa2kj-bcc1a100-d380-42cf-bccf-2e32badf7ac5.stats A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/a874fc23-2c2c-42b5-9fc0-f72919a7449d-m0.avro A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/snap-6097489456815673056-1-a874fc23-2c2c-42b5-9fc0-f72919a7449d.avro A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/snap-8332248857076198293-1-03e34f74-1a29-4e53-ab2c-6f2c507d534d.avro A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_v3/trino_variant/metadata/version-hint.text A testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-variant.test M tests/query_test/test_iceberg.py M tests/util/iceberg_metadata_util.py 59 files changed, 896 insertions(+), 65 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/24521/1 -- To view, visit http://gerrit.cloudera.org:8080/24521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie2f8a7c9b1d4e5f6a0c3b8d7e9f1a2b4c6d8e0f1 Gerrit-Change-Number: 24521 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
