Dejan Krakovic created SPARK-61304:
--------------------------------------
Summary: Decide whether to reject FILE in aggregates that
deduplicate or hash their input
Key: SPARK-61304
URL: https://issues.apache.org/jira/browse/SPARK-61304
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Dejan Krakovic
Follow-up of SPARK-60386 (https://github.com/apache/spark/pull/59640).
FileType is not comparable, but aggregates that deduplicate or hash their input
still accept a FILE at analysis: count(DISTINCT f), collect_set(f),
approx_count_distinct(f), mode(f) and collect_union. They cannot run before
FILE has a physical representation. Decide, with the physical representation,
whether to reject them at analysis like the other operations that need equality.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
