Hemanth Boyina created SPARK-60439:
--------------------------------------
Summary: Support DataSource V2 pushdown for aggregates with FILTER
clauses
Key: SPARK-60439
URL: https://issues.apache.org/jira/browse/SPARK-60439
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Hemanth Boyina
Spark supports aggregate pushdown, but aggregates with a FILTER clause are
currently evaluated in Spark rather than the data source.
For example:
SELECT
SUM(amount) FILTER (WHERE status = 'paid'),
COUNT(*) FILTER (WHERE status = 'failed'),
COUNT(*)
FROM sales;
Allow supported data sources to compute these measures by translating each
filtered aggregate into an equivalent aggregate over CASE WHEN. This reduces
input rows transferred to Spark while preserving each measure’s condition.
Initial support covers deterministic filters on non-distinct MIN, MAX, SUM,
AVG, and single-input COUNT, including COUNT(*). Require complete aggregate
pushdown and retain Spark execution for unsupported cases. No new public API is
needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
