Venkata krishnan Sowrirajan created SPARK-35536:
---------------------------------------------------
Summary: Cancel finalizing the shuffle merge if the stage is
cancelled while waiting until shuffle merge finalize wait time.
Key: SPARK-35536
URL: https://issues.apache.org/jira/browse/SPARK-35536
Project: Spark
Issue Type: Sub-task
Components: Shuffle, Spark Core
Affects Versions: 3.1.0
Reporter: Venkata krishnan Sowrirajan
Currently we are not handling the case of stage cancellation and if the stage
is cancelled while the stage is about to get merge finalized.
Stage completion will wait for shuffle merge finalize wait time so that all the
shuffle merge requests are drained but during this time the stage can be
cancelled, if this happens then we need to let all the shuffle services to
remove the merged shuffle data for the corresponding shuffle ID.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
