[
https://issues.apache.org/jira/browse/SPARK-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831117#comment-17831117
]
Asif edited comment on SPARK-26708 at 3/27/24 12:54 AM:
--------------------------------------------------------
Towards that please take a look at ticket & PR:
https://issues.apache.org/jira/browse/SPARK-45959
and the PR associated with it.
Though that PR primarily deals with aggressive collapse of projects at the end
of analysis . But it also as part of fix, uses enhanced cached plan lookup and
thus results in the above behaviour.
was (Author: ashahid7):
Towards that please take a look at ticket & PR:
[https://issues.apache.org/jira/browse/SPARK-45959|https://issues.apache.org/jira/browse/SPARK-45959]
> Incorrect result caused by inconsistency between a SQL cache's cached RDD and
> its physical plan
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-26708
> URL: https://issues.apache.org/jira/browse/SPARK-26708
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Xiao Li
> Assignee: Wei Xue
> Priority: Blocker
> Labels: correctness
> Fix For: 2.4.1, 3.0.0
>
>
> When performing non-cascading cache invalidation, {{recache}} is called on
> the other cache entries which are dependent on the cache being invalidated.
> It leads to the the physical plans of those cache entries being re-compiled.
> For those cache entries, if the cache RDD has already been persisted, chances
> are there will be inconsistency between the data and the new plan. It can
> cause a correctness issue if the new plan's {{outputPartitioning}} or
> {{outputOrdering}} is different from the that of the actual data, and
> meanwhile the cache is used by another query that asks for specific
> {{outputPartitioning}} or {{outputOrdering}} which happens to match the new
> plan but not the actual data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
