Dejan Krakovic created SPARK-60386:
--------------------------------------
Summary: Enforce FILE EXTERNAL in table columns and reject FILE
where it would need an ordering
Key: SPARK-60386
URL: https://issues.apache.org/jira/browse/SPARK-60386
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 4.4.0
Reporter: Dejan Krakovic
Part 2 of the FileType SPIP (SPARK-59132), building on the Catalyst type added
in SPARK-60092.
The SPIP requires the EXTERNAL keyword when a FILE column is declared in a
table, and gives FileType no defined ordering. This ticket enforces both at
analysis time.
Table columns:
- A FILE in a table column definition must be FILE EXTERNAL, including when
nested in a struct, array or map. This covers CREATE TABLE, REPLACE TABLE,
ALTER TABLE ADD COLUMNS and ALTER TABLE
REPLACE COLUMNS, for V1 and V2 tables (FILE_TYPE_MISSING_QUALIFIER).
- CREATE TABLE AS SELECT cannot store a FILE that has no qualifier
(FILE_TYPE_UNKNOWN_WRITE).
- A FILE column cannot be a partition or clustering column
(INVALID_PARTITION_COLUMN_DATA_TYPE).
Routines:
- Following the SPIP, functions declare a plain FILE. A FILE EXTERNAL in the
parameter or return types of a SQL function is rejected
(FILE_TYPE_IN_ROUTINE_SIGNATURE_UNSUPPORTED_QUALIFIER), and
so is a FILE in a map key of its return type
(FILE_TYPE_IN_ROUTINE_SIGNATURE_UNSUPPORTED_MAP_KEY).
- The same applies to the return type of UDFs, both when they are registered
and when they are used in a query.
No ordering:
- FILE is rejected as a map key (INVALID_MAP_KEY_TYPE), a grouping expression
(GROUP_EXPRESSION_TYPE_IS_NOT_ORDERABLE), in INTERSECT, EXCEPT and DISTINCT
(UNSUPPORTED_FEATURE.SET_OPERATION_ON_FILE_TYPE), as a repartition expression
(UNSUPPORTED_FEATURE.PARTITION_BY_FILE), and in hash functions
(DATATYPE_MISMATCH.HASH_FILE_TYPE).
- Join keys, comparisons and sorting are already rejected because FileType is
not orderable (INVALID_ORDERING_TYPE). This ticket adds tests for them.
The checks for set operations and repartitioning apply to both the fixed-point
analyzer and the single-pass resolver. The FILE type remains gated by
spark.sql.fileType.enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
