Gengliang Wang created SPARK-61906:
--------------------------------------
Summary: Let native data sources report the partitioning and the
ordering of their data
Key: SPARK-61906
URL: https://issues.apache.org/jira/browse/SPARK-61906
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
A native data source cannot report how its data is partitioned or sorted
({{SupportsReportPartitioning}}, {{SupportsReportOrdering}}). So Spark shuffles
and sorts the output of a native scan whenever a join, an aggregate or a window
needs it, and cannot plan a storage-partitioned join over native scans.
Proposal: let a reader report the partitioning and the ordering of its scan,
with functions appended to {{SparkDsLibrary}} within version 1 of the interface
({{spark_datasource.h}}), and expose them in the Rust SDK (SPARK-61282).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]