Juliusz Sompolski created SPARK-58107:
-----------------------------------------
Summary: LocalCheckpoint can be inconsistent because local
executor BMs can store inconsistent blocks from task/stage retries
Key: SPARK-58107
URL: https://issues.apache.org/jira/browse/SPARK-58107
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.3.0
Reporter: Juliusz Sompolski
A locally-checkpointed RDD partition can be materialized by more than one
successful task attempt — from speculation, or a zombie task of a superseded
stage attempt during retries. When the computation is non-deterministic, the
attempts produce different bytes for the same RDDBlockId. The BlockManager
keeps both copies (the master's blockLocations just appends locations, with no
version/attempt id) and a reader picks one at random, so two reads of "the
same" local checkpoint can disagree and it is undetected.
This particularly affects Delta's MERGE source materialization, which
localCheckpoints the source DataFrame (a DISK_ONLY serialized level) to cut
lineage before the multi-job MERGE. A non-deterministic source (e.g. a source
read that isn't stable across attempts) can then be observed inconsistently
across the MERGE's passes, surfacing as invariant/row-count violations in the
written data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
