Gengliang Wang created SPARK-61906:
--------------------------------------

             Summary: Let native data sources report the partitioning and the 
ordering of their data
                 Key: SPARK-61906
                 URL: https://issues.apache.org/jira/browse/SPARK-61906
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


A native data source cannot report how its data is partitioned or sorted 
({{SupportsReportPartitioning}}, {{SupportsReportOrdering}}). So Spark shuffles 
and sorts the output of a native scan whenever a join, an aggregate or a window 
needs it, and cannot plan a storage-partitioned join over native scans.

Proposal: let a reader report the partitioning and the ordering of its scan, 
with functions appended to {{SparkDsLibrary}} within version 1 of the interface 
({{spark_datasource.h}}), and expose them in the Rust SDK (SPARK-61282).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to