[
https://issues.apache.org/jira/browse/SPARK-60386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60386:
-----------------------------------
Labels: pull-request-available (was: )
> Enforce FILE EXTERNAL in table columns and reject FILE where it would need an
> ordering
> --------------------------------------------------------------------------------------
>
> Key: SPARK-60386
> URL: https://issues.apache.org/jira/browse/SPARK-60386
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 4.4.0
> Reporter: Dejan Krakovic
> Priority: Major
> Labels: pull-request-available
>
> Part 2 of the FileType SPIP (SPARK-59132), building on the Catalyst type
> added in SPARK-60092.
> The SPIP requires the EXTERNAL keyword when a FILE column is declared in a
> table, and gives FileType no defined ordering. This ticket enforces both at
> analysis time.
> Table columns:
> - A FILE in a table column definition must be FILE EXTERNAL, including when
> nested in a struct, array or map. This covers CREATE TABLE, REPLACE TABLE,
> ALTER TABLE ADD COLUMNS and ALTER TABLE
> REPLACE COLUMNS, for V1 and V2 tables (FILE_TYPE_MISSING_QUALIFIER).
> - CREATE TABLE AS SELECT cannot store a FILE that has no qualifier
> (FILE_TYPE_UNKNOWN_WRITE).
> - A FILE column cannot be a partition or clustering column
> (INVALID_PARTITION_COLUMN_DATA_TYPE).
> Routines:
> - Following the SPIP, functions declare a plain FILE. A FILE EXTERNAL in the
> parameter or return types of a SQL function is rejected
> (FILE_TYPE_IN_ROUTINE_SIGNATURE_UNSUPPORTED_QUALIFIER), and
> so is a FILE in a map key of its return type
> (FILE_TYPE_IN_ROUTINE_SIGNATURE_UNSUPPORTED_MAP_KEY).
> - The same applies to the return type of UDFs, both when they are registered
> and when they are used in a query.
> No ordering:
> - FILE is rejected as a map key (INVALID_MAP_KEY_TYPE), a grouping
> expression (GROUP_EXPRESSION_TYPE_IS_NOT_ORDERABLE), in INTERSECT, EXCEPT and
> DISTINCT
> (UNSUPPORTED_FEATURE.SET_OPERATION_ON_FILE_TYPE), as a repartition
> expression (UNSUPPORTED_FEATURE.PARTITION_BY_FILE), and in hash functions
> (DATATYPE_MISMATCH.HASH_FILE_TYPE).
> - Join keys, comparisons and sorting are already rejected because FileType
> is not orderable (INVALID_ORDERING_TYPE). This ticket adds tests for them.
> The checks for set operations and repartitioning apply to both the
> fixed-point analyzer and the single-pass resolver. The FILE type remains
> gated by spark.sql.fileType.enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
