[
https://issues.apache.org/jira/browse/IMPALA-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy reassigned IMPALA-15434:
------------------------------------------
Assignee: Zoltán Borók-Nagy
> [DOCS] Document the VARIANT data type and the variant_get() functions
> ---------------------------------------------------------------------
>
> Key: IMPALA-15434
> URL: https://issues.apache.org/jira/browse/IMPALA-15434
> Project: IMPALA
> Issue Type: Documentation
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> Impala 5.0 reads VARIANT columns of Iceberg tables (IMPALA-15052), hardened
> the decoder (IMPALA-15221) and now has VARIANT functions (IMPALA-15057).
> There is no VARIANT content in docs/ at all.
> Create docs/topics/impala_variant.xml ("VARIANT Data Type (Impala 5.0 or
> higher only)"), add a <topicref> after impala_varchar.xml under
> impala_datatypes.xml in docs/impala.ditamap and a keydef in
> docs/impala_keydefs.ditamap. Add a variant row to the "Supported Data Types
> for Iceberg Columns" table in impala_iceberg.xml — it currently
> ends at map, so readers assume VARIANT is unsupported.
> The page should cover:
> - Read-only support for top-level, unshredded VARIANT columns in Iceberg
> tables with Parquet data files, as written by engines such as Spark or Trino.
> Shredded values fail at scan time.
> - Values come back as JSON text: SQL NULL versus a variant null ('null'),
> quoted strings, base64 for BINARY, quoted DATE and TIMESTAMP, and
> "<unsupported-type>" for TIME, TIMESTAMPTZ and UUID. Clients see STRING; over
> JDBC it is VARCHAR with TYPE_NAME 'VARIANT'.
> - Functions (IMPALA-15057): variant_get(VARIANT, path) -> VARIANT
> (chainable), the typed variant_get(VARIANT, path, 'type'), the
> try_variant_get() forms, and variant_to_json(VARIANT) -> STRING. Document the
> path syntax, the coercion rules of the 3-arg form (widening, checked
> narrowing, string parsing, and DATE only from a real DATE
> value), and that any query with VARIANT expressions runs interpreted,
> without codegen.
> - Restrictions: VARIANT can't be used in CREATE TABLE, ALTER TABLE ADD COLUMN
> or CAST ("VARIANT type is not supported as a column type."); no INSERT into a
> table with a VARIANT column, so CTAS and INSERT ... SELECT fail; and VARIANT
> is rejected in ORDER BY, GROUP BY, SELECT DISTINCT, UNION/INTERSECT/EXCEPT,
> CASE/DECODE, analytic
> PARTITION BY and ORDER BY, aggregate DISTINCT and IS [NOT] NULL. The
> current message is "<clause> '<expr>' with type 'VARIANT' is not supported."
> UNION ALL, joins, inline views and WITH work.
> - A VARIANT nested inside an ARRAY, MAP or STRUCT can't be queried, though
> DESCRIBE works. VARIANT is not a complex type, so EXPAND_COMPLEX_TYPES
> doesn't control it and SELECT * always returns it.
> - Limits from IMPALA-15221: a value nested more than 100 levels deep fails; a
> DATE outside 0001-01-01..9999-12-31 or a TIMESTAMP outside
> 1400-01-01..9999-12-31 inside a VARIANT fails the query instead of returning
> NULL (this can happen with valid files from other engines); corrupt encodings
> fail with "Malformed variant ...".
> - COMPUTE STATS skips VARIANT columns. SHOW CREATE TABLE prints v VARIANT,
> but that output can't be re-run because VARIANT is rejected in DDL. HMS
> stores the column as struct<metadata:binary,value:binary>, which matters for
> interop.
> VARIANT is a non-reserved keyword, so no incompatible-changes entry and no
> reserved-words change are needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
