[
https://issues.apache.org/jira/browse/SPARK-55569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Malthe Borch updated SPARK-55569:
---------------------------------
Description:
When pivoting on a set of values, this is potentially a subset of all values.
Nonetheless, there's no predicate pushdown to the source.
{code:sql}
SELECT * FROM person
PIVOT (
SUM(age) AS a, AVG(class) AS c
FOR name IN ('John' AS john, 'Mike' AS mike)
);
{code}
In this example there is no predicate pushdown to filter the {{person}} table
on the specified names.
The workaround is to repeat the same set of values in a WHERE-clause before the
pivot clause, but this is redundant. In addition, I think it can be argued that
you expect the optimizer to do this.
was:
When pivoting on a set of values, this is potentially a subset of all values.
Nonetheless, there's no predicate pushdown to the source.
The workaround is to repeat the same set of values in a WHERE-clause before the
pivot clause, but this is redundant. In addition, I think it can be argued that
you expect the optimizer to do this.
> Filter pushdown in pivot aggregation
> ------------------------------------
>
> Key: SPARK-55569
> URL: https://issues.apache.org/jira/browse/SPARK-55569
> Project: Spark
> Issue Type: Wish
> Components: Optimizer
> Affects Versions: 4.1.1
> Reporter: Malthe Borch
> Priority: Major
>
> When pivoting on a set of values, this is potentially a subset of all values.
> Nonetheless, there's no predicate pushdown to the source.
> {code:sql}
> SELECT * FROM person
> PIVOT (
> SUM(age) AS a, AVG(class) AS c
> FOR name IN ('John' AS john, 'Mike' AS mike)
> );
> {code}
> In this example there is no predicate pushdown to filter the {{person}} table
> on the specified names.
> The workaround is to repeat the same set of values in a WHERE-clause before
> the pivot clause, but this is redundant. In addition, I think it can be
> argued that you expect the optimizer to do this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
