Tengfei Huang created SPARK-60548:
-------------------------------------
Summary: Handle shuffle checksum mismatches detected in stage
attempt 0
Key: SPARK-60548
URL: https://issues.apache.org/jira/browse/SPARK-60548
Project: Spark
Issue Type: Task
Components: Spark Core
Affects Versions: 4.2.0
Reporter: Tengfei Huang
When a shuffle checksum mismatch is detected while registering map output,
DAGScheduler rolls back the succeeding stages only if
Stage.maxChecksumMismatchedId < stageAttemptId. Stage.maxChecksumMismatchedId
is initialized to nextAttemptId, which is 0 when the stage is created, so a
mismatch reported by a task from stage attempt 0 never passes the guard and is
silently accepted. Consumers that already read the old output keep results
computed from it, which can lead to incorrect results.
There are two cases where a mismatch can be detected in attempt 0:
1. A shuffle map stage is recreated for an existing shuffle. After a job
finishes, its stages are cleaned up, but MapOutputTracker keeps the shuffle's
map statuses and their checksums. If some map output is later lost and a
subsequent job that depends on the same shuffle recomputes the missing
partitions, the new ShuffleMapStage starts at attempt 0. If the recomputed data
differs from the previous output, the checksum mismatch is detected in attempt
0 and no rollback happens.
2. A late task from attempt 0 replaces the output of a later attempt. When
attempt 0 of a shuffle map stage fails with a FetchFailed, its other running
tasks are not killed by default. If one of them finishes after attempt 1 has
recomputed the same partition, it replaces attempt 1's registered map output. A
changed checksum here is silently accepted, while the reverse order (a late
copy from attempt 1 replacing attempt 0's output) already triggers rollback.
Proposed fix: initialize Stage.maxChecksumMismatchedId to -1, so the first
checksum mismatch in attempt 0 goes through the existing rollback/abort
handling. Later mismatches in the same attempt are still deduplicated, and the
behavior for attempts 1 and later is unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
