Dejan Krakovic created SPARK-61302:
--------------------------------------
Summary: Decide whether FileType defines equality
Key: SPARK-61302
URL: https://issues.apache.org/jira/browse/SPARK-61302
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Dejan Krakovic
Follow-up of SPARK-60386 (https://github.com/apache/spark/pull/59640).
The SPIP (SPARK-59132) defines FileType as not comparable, because "it is not
clear what equality of two files should mean": a FILE cannot be a join,
grouping or sort key, be deduplicated or hashed, or be a map or partition key.
Users extract the uri instead. Decide whether FileType should ever define
equality (and hashing), and what it would mean, for example equality of uri, or
of uri and checksum. If it does, the checks added in SPARK-60386 are where it
would be relaxed.
Review thread: https://github.com/apache/spark/pull/59640#discussion_r4461444855
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
