Venkata krishnan Sowrirajan created SPARK-61465:
---------------------------------------------------

             Summary: Avoid map-output epoch bump on an empty host cleanup when 
all shuffles are reliably stored and push-based shuffle is on
                 Key: SPARK-61465
                 URL: https://issues.apache.org/jira/browse/SPARK-61465
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 4.3.1
            Reporter: Venkata krishnan Sowrirajan


In SPARK-59138, `handleExecutorLost` skips map-output cleanup when 
`ShuffleDriverComponents.supportsReliableStorage()` is true and no registered 
shuffle is unreliable. This matches the behavior before that change. With 
push-based shuffle on, a loss that names a worker host still runs 
`MapOutputTrackerMaster.removeOutputsOnHost(host, respectReliablyStored = 
true)` so the host's merge results are dropped.

If that host held no map output and no merge results, the pass returns 
`CleanupOutcome(metadataChanged = false, preservedReliable = false)`. 
`shouldBumpEpoch` is then true, so the map-output epoch advances. Every 
executor drops its cached map and merge statuses and refetches them from the 
driver, even though nothing changed. Before SPARK-59138 this path did nothing 
for these applications.

We can't just stop bumping on an empty pass. The tracker relies on "a no-op 
loss still bumps the epoch" for executor-failure fencing whenever unreliable 
shuffles are registered.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to