Andy Grove created SPARK-35093:
----------------------------------
Summary: [SQL] AQE columnar mismatch on exchange reuse
Key: SPARK-35093
URL: https://issues.apache.org/jira/browse/SPARK-35093
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.1.1, 3.0.2
Reporter: Andy Grove
With AQE enabled, AdaptiveSparkPlanExec will attempt to reuse exchanges that
are semantically equal.
This is done by comparing the canonicalized plan for two Exchange nodes to see
if they are the same.
Unfortunately this does not take into account the fact that two exchanges with
the same canonical plan might be replaced by a plugin in a way that makes them
not compatible. For example, a plugin could create one version with
supportsColumnar=true and another with supportsColumnar=false. It is not valid
to re-use exchanges if there is a supportsColumnar mismatch.
I have tested a fix for this and will put up a PR once I figure out how to
write the tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
