yew1eb commented on PR #3853: URL: https://github.com/apache/celeborn/pull/3853#issuecomment-6552481515
Thanks for the re-review. Both points are addressed in `27af422c` (suite passes 15/15 on Spark 3.5 and 4.0): - **Test GC fallback**: the suite now sets `spark.cleaner.referenceTracking=false`, so `awaitShufflesUnregistered` can only succeed through the query-end listener. - **Dropped stage with `spark.sql.exchange.reuse=false`**: confirmed — `stageCache` is only populated when exchange reuse is on, and a stage AQE drops from the final plan is not reachable from `executedPlan`. I've documented this as a remaining limitation in the config doc and the listener; the behavior fix (tracking shuffle ids at `registerShuffle` time) is better as a separate change. For the `UNKNOWN_APP_SHUFFLE_ID` race, I'll open the follow-up JIRA and link it here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
