Gengliang Wang created SPARK-61905:
--------------------------------------

             Summary: Push aggregates and top-N queries down to native data 
sources
                 Key: SPARK-61905
                 URL: https://issues.apache.org/jira/browse/SPARK-61905
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


Spark pushes the domains of filtered columns, a limit and the columns to read 
down to native data sources, but neither aggregates 
({{SupportsPushDownAggregates}}) nor top-N queries, an {{ORDER BY}} with a 
{{LIMIT}} ({{SupportsPushDownTopN}}). A library that can answer {{count(*)}}, 
{{min}} or {{max}} from the statistics of its data, or that reads from a system 
that can aggregate or sort, still returns all the rows.

Proposal: add both to the native data source interface ({{spark_datasource.h}}) 
within version 1, as functions appended to {{SparkDsLibrary}}, and expose them 
in the Rust SDK (SPARK-61282).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to