Zoltán Borók-Nagy created IMPALA-15434:
------------------------------------------
Summary: [DOCS] Document the VARIANT data type and the
variant_get() functions
Key: IMPALA-15434
URL: https://issues.apache.org/jira/browse/IMPALA-15434
Project: IMPALA
Issue Type: Documentation
Reporter: Zoltán Borók-Nagy
Impala 5.0 reads VARIANT columns of Iceberg tables (IMPALA-15052), hardened the
decoder (IMPALA-15221) and now has VARIANT functions (IMPALA-15057). There is
no VARIANT content in docs/ at all.
Create docs/topics/impala_variant.xml ("VARIANT Data Type (Impala 5.0 or higher
only)"), add a <topicref> after impala_varchar.xml under impala_datatypes.xml
in docs/impala.ditamap and a keydef in docs/impala_keydefs.ditamap. Add a
variant row to the "Supported Data Types for Iceberg Columns" table in
impala_iceberg.xml — it currently
ends at map, so readers assume VARIANT is unsupported.
The page should cover:
- Read-only support for top-level, unshredded VARIANT columns in Iceberg tables
with Parquet data files, as written by engines such as Spark or Trino. Shredded
values fail at scan time.
- Values come back as JSON text: SQL NULL versus a variant null ('null'),
quoted strings, base64 for BINARY, quoted DATE and TIMESTAMP, and
"<unsupported-type>" for TIME, TIMESTAMPTZ and UUID. Clients see STRING; over
JDBC it is VARCHAR with TYPE_NAME 'VARIANT'.
- Functions (IMPALA-15057): variant_get(VARIANT, path) -> VARIANT (chainable),
the typed variant_get(VARIANT, path, 'type'), the try_variant_get() forms, and
variant_to_json(VARIANT) -> STRING. Document the path syntax, the coercion
rules of the 3-arg form (widening, checked narrowing, string parsing, and DATE
only from a real DATE
value), and that any query with VARIANT expressions runs interpreted, without
codegen.
- Restrictions: VARIANT can't be used in CREATE TABLE, ALTER TABLE ADD COLUMN
or CAST ("VARIANT type is not supported as a column type."); no INSERT into a
table with a VARIANT column, so CTAS and INSERT ... SELECT fail; and VARIANT is
rejected in ORDER BY, GROUP BY, SELECT DISTINCT, UNION/INTERSECT/EXCEPT,
CASE/DECODE, analytic
PARTITION BY and ORDER BY, aggregate DISTINCT and IS [NOT] NULL. The current
message is "<clause> '<expr>' with type 'VARIANT' is not supported." UNION ALL,
joins, inline views and WITH work.
- A VARIANT nested inside an ARRAY, MAP or STRUCT can't be queried, though
DESCRIBE works. VARIANT is not a complex type, so EXPAND_COMPLEX_TYPES doesn't
control it and SELECT * always returns it.
- Limits from IMPALA-15221: a value nested more than 100 levels deep fails; a
DATE outside 0001-01-01..9999-12-31 or a TIMESTAMP outside
1400-01-01..9999-12-31 inside a VARIANT fails the query instead of returning
NULL (this can happen with valid files from other engines); corrupt encodings
fail with "Malformed variant ...".
- COMPUTE STATS skips VARIANT columns. SHOW CREATE TABLE prints v VARIANT, but
that output can't be re-run because VARIANT is rejected in DDL. HMS stores the
column as struct<metadata:binary,value:binary>, which matters for interop.
VARIANT is a non-reserved keyword, so no incompatible-changes entry and no
reserved-words change are needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
