Zoltán Borók-Nagy created IMPALA-15432:
------------------------------------------
Summary: [DOCS] Fix and extend the EXPAND_COMPLEX_TYPES
default-change documentation
Key: IMPALA-15432
URL: https://issues.apache.org/jira/browse/IMPALA-15432
Project: IMPALA
Issue Type: Documentation
Reporter: Zoltán Borók-Nagy
Assignee: Zoltán Borók-Nagy
IMPALA-15054 changed the default of EXPAND_COMPLEX_TYPES from FALSE to TRUE and
documented it. Three problems remain.
(a) The incompatible-changes entry has a wrong example.
impala_incompatible_changes.xml, concept incompatible_changes_500x, says the
failure case is "a complex column whose type is not supported in the select
list (for example a STRUCT nested inside a collection)". Mixed complex types in
the select list have been supported since Impala
4.3 (IMPALA-9551), so that is not a failure case. The cases that really fail
under the new default:
- CREATE TABLE AS SELECT * and INSERT ... SELECT * from a table with complex
columns: "... because the column '<c>' has a complex type '<t>' and Impala
doesn't support inserting complex types" (InsertStmt)
- SELECT DISTINCT * and GROUP BY over an expanded collection: "SELECT DISTINCT
expression '<e>' with type '<t>' is not supported." (Type.throwIfNotComparable)
- UNION, INTERSECT or EXCEPT without ALL: "UNION, EXCEPT and INTERSECT with
type '<t>' is not supported." (SetOperationStmt)
- SELECT * ... ORDER BY when a STRUCT contains a collection: "Sorting is not
supported if the select list contains collection(s) nested in struct(s)."
(QueryStmt.checkTypeValidityForSorting)
- STRUCT columns in tables that are not Parquet or ORC, e.g. Paimon ROW
columns or RCFile/SequenceFile partitions: "Querying STRUCT is only supported
for ORC and Parquet file formats." (SlotRef)
Also worth a sentence: unnest joins such as SELECT * FROM t, t.arr now return
the parent's complex columns too, so clients that read columns by position are
affected. Keep the SET EXPAND_COMPLEX_TYPES=false workaround and add "list the
columns explicitly" as the alternative.
(b) The option page doesn't record the version.
impala_expand_complex_types.xml says only "Default: TRUE / Added in: Impala
4.2". Add "The default changed from FALSE to TRUE in Impala 5.0" (the pattern
used by impala_default_file_format.xml) and xref the 5.0.x incompatible-changes
entry.
(c) Two stale spots in impala_complex_types.xml.
- Around line 2109, in the view example: "SELECT * would bring back all the
data, unlike SELECT * queries on the original table with complex type columns".
SELECT * on the base table now returns the complex columns too.
- The "not supported in the select list" list (STRUCT<a: ARRAY<INT>>,
ARRAY<STRUCT<i: INT>>, MAP<INT, STRUCT<s: STRING>>) and the sentence above it
about collections embedded in structs. Stale since 4.3, and now actively
misleading, because SELECT * expands exactly these columns by default. The real
remaining limits: no sorting of
collections nested in structs, no DISTINCT or set operations over
collections, and STRUCT only for Parquet/ORC.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
