[
https://issues.apache.org/jira/browse/SPARK-60548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60548:
-----------------------------------
Labels: pull-request-available (was: )
> Handle shuffle checksum mismatches detected in stage attempt 0
> --------------------------------------------------------------
>
> Key: SPARK-60548
> URL: https://issues.apache.org/jira/browse/SPARK-60548
> Project: Spark
> Issue Type: Task
> Components: Spark Core
> Affects Versions: 4.2.0
> Reporter: Tengfei Huang
> Priority: Major
> Labels: pull-request-available
>
> When a shuffle checksum mismatch is detected while registering map output,
> DAGScheduler rolls back the succeeding stages only if
> Stage.maxChecksumMismatchedId < stageAttemptId. Stage.maxChecksumMismatchedId
> is initialized to nextAttemptId, which is 0 when the stage is created, so a
> mismatch reported by a task from stage attempt 0 never passes the guard and
> is silently accepted. Consumers that already read the old output keep results
> computed from it, which can lead to incorrect results.
> There are two cases where a mismatch can be detected in attempt 0:
> 1. A shuffle map stage is recreated for an existing shuffle. After a job
> finishes, its stages are cleaned up, but MapOutputTracker keeps the shuffle's
> map statuses and their checksums. If some map output is later lost and a
> subsequent job that depends on the same shuffle recomputes the missing
> partitions, the new ShuffleMapStage starts at attempt 0. If the recomputed
> data differs from the previous output, the checksum mismatch is detected in
> attempt 0 and no rollback happens.
> 2. A late task from attempt 0 replaces the output of a later attempt. When
> attempt 0 of a shuffle map stage fails with a FetchFailed, its other running
> tasks are not killed by default. If one of them finishes after attempt 1 has
> recomputed the same partition, it replaces attempt 1's registered map output.
> A changed checksum here is silently accepted, while the reverse order (a late
> copy from attempt 1 replacing attempt 0's output) already triggers rollback.
> Proposed fix: initialize Stage.maxChecksumMismatchedId to -1, so the first
> checksum mismatch in attempt 0 goes through the existing rollback/abort
> handling. Later mismatches in the same attempt are still deduplicated, and
> the behavior for attempts 1 and later is unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
