[
https://issues.apache.org/jira/browse/IMPALA-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100282#comment-18100282
]
ASF subversion and git services commented on IMPALA-15052:
----------------------------------------------------------
Commit 480cdc593de804634e52b79f8eec4a75709c9ee0 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=480cdc593 ]
IMPALA-15052: Add read support for unshredded VARIANT values
Builds on the backend VARIANT decoding from IMPALA-15067 to enable
end-to-end reading of unshredded VARIANT columns from Iceberg tables
stored as Parquet. VARIANT is read-only: it can be scanned and rendered
as JSON, but it cannot be created or written, and it can only appear in
queries as a pass-through column reference.
Type system:
- TYPE_VARIANT added across thrift (TPrimitiveType/TTypeNodeType), the
BE (PrimitiveType/ColumnType, 24-byte slot = two StringValues) and the
FE (VariantType extending Type with struct-like metadata/value fields)
- An unshredded VARIANT is modeled as a two-child group: "metadata" and
"value", both BINARY.
Read path (Parquet):
- VariantColumnReader (extends StructColumnReader) reads the variant
group's metadata + value children. The physical layout matches a
two-field struct, so ParquetSchemaResolver resolves VARIANT like
STRUCT.
- A malformed variant group whose Parquet child count does not match the
catalog (i.e. not exactly metadata + value) fails with a runtime
error.
- The reader is excluded from late materialization (it inherits the
struct SkipRows() restriction) so a filtered scan never calls
SkipRows() on it.
- FE descriptors: the Analyzer builds a children tuple with
metadata/value slots for each VARIANT column.
Result display:
- VARIANT columns are serialized to their JSON representation in query
output (hs2-util, query-result-set), via the backend VariantSlotToJson
helper. BINARY fields inside a variant are base64-encoded in the JSON
(as for complex types; matches Trino).
- A VARIANT value that fails to decode (corruption) fails the query,
instead of being silently substituted with a wrong value.
- To clients (HS2 result metadata, GetColumns, beeswax) a VARIANT column
is reported as STRING/VARCHAR, like the other complex types.
NOTE: This may vary, as client interfaces may add native support for
VARIANT types in the future.
DDL and analysis (VARIANT is read-only):
- VARIANT is a non-reserved keyword: it is recognized by the parser as a
type, but existing identifiers named "variant" keep working. It is
rejected as a user-specified column type or CAST target (CREATE/ALTER
TABLE, CAST).
Unsupported operations by design
- ORDER BY, GROUP BY, SELECT DISTINCT, UNION/INTERSECT/EXCEPT, CASE/DECODE
analytic PARTITION BY/ORDER BY, aggregate DISTINCT
Currently not supported operations
- IS NULL, COMPUTE STATS, INSERT, CASE
Supported operations
- UNION ALL of a variant column is supported.
- Only a top-level VARIANT column can be queried. A VARIANT nested
inside another complex type (ARRAY<VARIANT>, MAP<..,VARIANT>,
STRUCT<..VARIANT..>, and deeper combinations) is rejected. Such a
table still loads; only queries that reference the nested VARIANT fail
Iceberg / HMS:
- IcebergSchemaConverter maps the Iceberg VARIANT type id to
TYPE_VARIANT.
- HMS has no 'variant' column type, so Type.toHiveMetastoreType()
stores VARIANT as its underlying struct<metadata:binary,value:binary>
(matching Iceberg's Hive catalog and Trino). Impala still reads the
real VARIANT type from the Iceberg metadata; the HMS FieldSchema is
only used for Hive interop.
- DESCRIBE and DESCRIBE FORMATTED show the user-facing 'variant' type,
not the HMS struct (DescribeResultFactory builds the formatted output
from the Impala type).
Codegen:
- Codegen is disabled for queries that scan a VARIANT column (currently
query-wide; see TODO in HdfsScanNode), and
Tuple::CodegenMaterializeExprs falls back to the interpreted path for
VARIANT slots. The interpreted infrastructure treats VARIANT like
STRUCT to avoid crashes.
Forward compatibility:
- No user-facing variant_get()/variant_to_json() builtin is registered
yet. These will be added once VARIANT is a first-class expression type
(a VariantVal ABI that lets functions accept and return VARIANT).
Testing:
- New e2e test (iceberg-v3-variant.test) covering JSON rendering of
primitives/strings/arrays/objects, filtered scans, TOP-N, UNION ALL,
spilling joins and a spilling sort, slot/tuple unification, and the
negative analysis cases above, against a Trino-written Iceberg V3
table (trino_variant).
- The same test also covers the nested-VARIANT rejections against a
second Trino-written table (trino_nested_variant) that carries VARIANT
nested in ARRAY/MAP/STRUCT (and ARRAY<STRUCT<VARIANT>>), plus a
top-level VARIANT column to confirm that case still works. It also
checks DESCRIBE and DESCRIBE FORMATTED rendering of the VARIANT and
nested-VARIANT columns (describing such a table is always allowed).
- trino_variant is bulk-loaded during data loading, so tests read the
shared functional_parquet.trino_variant table.
- JdbcTest covers the JDBC client-facing presentation (VARCHAR,
TYPE_NAME 'VARIANT', JSON values).
- ParserTest confirms VARIANT parses as a type and remains usable as an
identifier (non-reserved keyword).
Change-Id: Ie2f8a7c9b1d4e5f6a0c3b8d7e9f1a2b4c6d8e0f1
Assisted-by: Claude Opus 4.6 <[email protected]>
Assisted-by: Claude Opus 4.8 (1M context) <[email protected]>
Reviewed-on: http://gerrit.cloudera.org:8080/24521
Reviewed-by: Csaba Ringhofer <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Read unshredded VARIANT values
> ------------------------------
>
> Key: IMPALA-15052
> URL: https://issues.apache.org/jira/browse/IMPALA-15052
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend, Catalog, Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg, impala-iceberg-active-backlog
>
> Add support for reading unshredded VARIANT values.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
