[
https://issues.apache.org/jira/browse/SPARK-58107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58107:
-----------------------------------
Labels: pull-request-available (was: )
> LocalCheckpoint can be inconsistent because local executor BMs can store
> inconsistent blocks from task/stage retries
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58107
> URL: https://issues.apache.org/jira/browse/SPARK-58107
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.3.0
> Reporter: Juliusz Sompolski
> Priority: Major
> Labels: pull-request-available
>
> A locally-checkpointed RDD partition can be materialized by more than one
> successful task attempt — from speculation, or a zombie task of a superseded
> stage attempt during retries. When the computation is non-deterministic, the
> attempts produce different bytes for the same RDDBlockId. The BlockManager
> keeps both copies (the master's blockLocations just appends locations, with
> no version/attempt id) and a reader picks one at random, so two reads of "the
> same" local checkpoint can disagree and it is undetected.
> This particularly affects Delta's MERGE source materialization, which
> localCheckpoints the source DataFrame (a DISK_ONLY serialized level) to cut
> lineage before the multi-job MERGE. A non-deterministic source (e.g. a source
> read that isn't stable across attempts) can then be observed inconsistently
> across the MERGE's passes, surfacing as invariant/row-count violations in the
> written data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
