venkata91 commented on PR #3853: URL: https://github.com/apache/celeborn/pull/3853#issuecomment-6563651502
non-blocking suggestion: for Spark 4, could we also collect IDs from `adaptivePlan.context.shuffleIds`, then use the existing Celeborn-registration filter and cleanup path? Spark records these when stages successfully materialize, independently of exchange reuse, so this should cover stages later dropped from the final plan. Source: https://github.com/apache/spark/blob/v4.0.3/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala#L306-L316 A dropped-stage test with `spark.sql.exchange.reuse=false` and GC cleanup disabled would cover it. Spark 3 doesn't have this collection and needs separate tracking. I'm fine with handling this here or in a follow-up PR, whichever you prefer. _Drafted by an AI agent for @venkata91._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
