Gengliang Wang created SPARK-61905:
--------------------------------------
Summary: Push aggregates and top-N queries down to native data
sources
Key: SPARK-61905
URL: https://issues.apache.org/jira/browse/SPARK-61905
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
Spark pushes the domains of filtered columns, a limit and the columns to read
down to native data sources, but neither aggregates
({{SupportsPushDownAggregates}}) nor top-N queries, an {{ORDER BY}} with a
{{LIMIT}} ({{SupportsPushDownTopN}}). A library that can answer {{count(*)}},
{{min}} or {{max}} from the statistics of its data, or that reads from a system
that can aggregate or sort, still returns all the rows.
Proposal: add both to the native data source interface ({{spark_datasource.h}})
within version 1, as functions appended to {{SparkDsLibrary}}, and expose them
in the Rust SDK (SPARK-61282).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]