Ziqi Liu created SPARK-49982:
--------------------------------
Summary: Negative AQE caching in SQL cache
Key: SPARK-49982
URL: https://issues.apache.org/jira/browse/SPARK-49982
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.0
Reporter: Ziqi Liu
When we use a cached an AQE plan, it will do `cachedPlan.execute` to build the
RDD, which will execute all AQE stages except the result stage. If any of them
failed, the failure will be cached by lazy RDD val. So the next any when we
reuse that cached plan (even by a totally irrelevant caller) it will fail
immediately.
We need to re-cache the AQE plan upon failure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
