[
https://issues.apache.org/jira/browse/SPARK-49982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ziqi Liu updated SPARK-49982:
-----------------------------
Description:
When we use a cached an AQE plan, it will do `cachedPlan.execute` to build the
RDD, which will execute all AQE stages except the result stage. If any of them
failed, the failure will be cached by lazy RDD val. So the next time when we
reuse that cached plan (even by a totally irrelevant caller) it will fail
immediately.
We need to re-cache the AQE plan upon failure.
was:
When we use a cached an AQE plan, it will do `cachedPlan.execute` to build the
RDD, which will execute all AQE stages except the result stage. If any of them
failed, the failure will be cached by lazy RDD val. So the next any when we
reuse that cached plan (even by a totally irrelevant caller) it will fail
immediately.
We need to re-cache the AQE plan upon failure.
> Negative AQE caching in SQL cache
> ---------------------------------
>
> Key: SPARK-49982
> URL: https://issues.apache.org/jira/browse/SPARK-49982
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Ziqi Liu
> Priority: Major
> Labels: pull-request-available
>
> When we use a cached an AQE plan, it will do `cachedPlan.execute` to build
> the RDD, which will execute all AQE stages except the result stage. If any of
> them failed, the failure will be cached by lazy RDD val. So the next time
> when we reuse that cached plan (even by a totally irrelevant caller) it will
> fail immediately.
> We need to re-cache the AQE plan upon failure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
