[
https://issues.apache.org/jira/browse/SPARK-60439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60439:
-----------------------------------
Labels: pull-request-available (was: )
> Support DataSource V2 pushdown for aggregates with FILTER clauses
> -----------------------------------------------------------------
>
> Key: SPARK-60439
> URL: https://issues.apache.org/jira/browse/SPARK-60439
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Hemanth Boyina
> Priority: Major
> Labels: pull-request-available
>
> Spark supports aggregate pushdown, but aggregates with a FILTER clause are
> currently evaluated in Spark rather than the data source.
>
> For example:
>
>
>
>
>
> SELECT
>
>
> SUM(amount) FILTER (WHERE status = 'paid'),
>
>
> COUNT(*) FILTER (WHERE status = 'failed'),
>
>
> COUNT(*)
>
>
> FROM sales;
>
>
>
>
>
> Allow supported data sources to compute these measures by translating each
> filtered aggregate into an equivalent aggregate over CASE WHEN. This reduces
> input rows transferred to Spark while preserving each measure’s condition.
>
>
>
>
>
>
> Initial support covers deterministic filters on non-distinct MIN, MAX, SUM,
> AVG, and single-input COUNT, including COUNT(*). Require complete aggregate
> pushdown and retain Spark execution for unsupported cases. No new public API
> is needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
