Thomas Graves created SPARK-32430:
-------------------------------------
Summary: Allow plugins to inject rules into AQE query stage
preparation
Key: SPARK-32430
URL: https://issues.apache.org/jira/browse/SPARK-32430
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.0
Reporter: Thomas Graves
Provide a generic mechanism for plugins to inject rules into the AQE "query
prep" stage that happens before query stage creation.
this goes along with https://issues.apache.org/jira/browse/SPARK-32332 where
the current AQE implementation doesn't allow for users to properly extend it
for columnar processing.
The issue here is that we create new query stages but we do not have access to
the parent plan of the new query stage so certain things can not be determined
because you have to know what the parent did. With this change it would allow
you to add TAGs to be able to figure out what is going on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
