yew1eb commented on PR #3853:
URL: https://github.com/apache/celeborn/pull/3853#issuecomment-6567620870
@venkata91 Thanks — done in f08309b.
Confirmed: `context.shuffleIds` is written from the stage materialization
callback in `AdaptiveSparkPlanExec` (`stage.plan.collect { case s:
ShuffleExchangeLike => context.shuffleIds.put(s.shuffleId, true) }`), so it is
independent of `spark.sql.exchange.reuse`, and it is still present in 4.1/4.2.
Since this module also compiles against Spark 3.x, I resolve the accessor
reflectively (same pattern as `isMaterialized`) and merge those ids with the
stage-cache/tree-walk results. The existing `isAppShuffleRegistered` filter and
the idempotent `removeShuffle` keep the union safe; Spark 3 keeps the
documented GC fallback since the collection does not exist there.
Test: added the dropped-stage case with `spark.sql.exchange.reuse=false`
(the suite already disables `spark.cleaner.referenceTracking`), gated to Spark
4.
- Spark 4.0.3: `CelebornShuffleCleanupSuite` 16/16 passed.
- Spark 3.5.9: 15/15 passed, the new case canceled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
