[
https://issues.apache.org/jira/browse/SPARK-60084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60084:
-----------------------------------
Labels: pull-request-available (was: )
> PlanMerger loses a reused filter attribute that an earlier merge folded into
> another filter
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-60084
> URL: https://issues.apache.org/jira/browse/SPARK-60084
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Peter Toth
> Priority: Major
> Labels: pull-request-available
>
> With filter propagation through a join, a merged subplan can end up with an
> aggregate {{FILTER}} on an attribute its child doesn't output. The query then
> fails with {{INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND}}.
> It needs both of these configs, both are off by default:
> *
> {{spark.sql.optimizer.mergeSubplans.filterPropagation.symmetricFilterPropagation.enabled=true}}
> *
> {{spark.sql.optimizer.mergeSubplans.filterPropagation.throughJoin.enabled=true}}
> Repro:
> {code:sql}
> CREATE TABLE t USING parquet AS SELECT * FROM VALUES
> (1, 1, 10), (1, 2, 3), (2, 1, 7), (2, 2, 9), (3, 1, 9), (3, 2, 6) AS v(a,
> b, c);
> SELECT
> (SELECT sum(l.a) FROM t l LEFT JOIN t r ON l.a = r.a WHERE l.b = 1),
> (SELECT max(l.a) FROM t l LEFT JOIN t r ON l.a = r.a WHERE l.b = 2 AND
> coalesce(r.c, 10) > 5),
> (SELECT count(*) FROM t l LEFT JOIN t r ON l.a = r.a WHERE l.b = 2);
> {code}
> Expected {{12, 3, 6}}, but it fails with:
> {noformat}
> [INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND] Could not find propagatedFilter_3#179 in
> [sum#141L,max#182,count#145L,a#6,propagatedFilter_5#181,propagatedFilter_4#180].
> SQLSTATE: XX000
> {noformat}
> What happens:
> # Merging the 2nd subquery into the 1st creates a filter attribute for {{l.b
> = 2}} on the left side of the join.
> # The 2nd subquery's post-join {{coalesce(r.c, 10) > 5}} folds that attribute
> into a new {{AND}} alias. Only the new alias is carried up to the aggregate.
> # The 3rd subquery reuses the {{l.b = 2}} attribute. {{PlanMerger}} flags a
> reused attribute as already present in the {{Project}}s above, so it doesn't
> append it there. Here it isn't present, so the 3rd subquery's {{FILTER}}
> references a missing attribute.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
