Dejan Krakovic created SPARK-61299:
--------------------------------------
Summary: Declare equality and ordering support per data type and
share the checks for MAP, VARIANT and FILE
Key: SPARK-61299
URL: https://issues.apache.org/jira/browse/SPARK-61299
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Dejan Krakovic
Follow-up of SPARK-60386 (https://github.com/apache/spark/pull/59640).
MAP, VARIANT and FILE are each rejected in operations that need equality (set
operations, DISTINCT, grouping, hashing, map, repartition and partition keys)
by per-type copies of the same checks in CheckAnalysis, TypeUtils, ExprUtils,
hash expressions and the single-pass OperatorWithUncomparableTypeValidator. The
copies have started to drift: FILE covers the REBALANCE hint and
DeduplicateWithinWatermark keys, VARIANT does not. Ordering is already a type
property (RowOrdering.isOrderable), but there is no equivalent for equality.
Proposal:
- Add a type-level way to declare whether a data type supports equality and
ordering (for example a trait or a method on DataType), and derive these checks
from it instead of listing types.
- Share the checks for MAP, VARIANT and FILE, and extend the VARIANT checks to
REBALANCE and DeduplicateWithinWatermark.
- Move the FILE-specific checks from TypeUtils into a dedicated object.
Review threads:
https://github.com/apache/spark/pull/59640#discussion_r4353281782,
https://github.com/apache/spark/pull/59640#discussion_r4449840766,
https://github.com/apache/spark/pull/59640#discussion_r4449844983
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
