yew1eb opened a new pull request, #3853: URL: https://github.com/apache/celeborn/pull/3853
### What changes were proposed in this pull request? Add an opt-in configuration `celeborn.client.spark.sql.queryEndShuffleCleanup.enabled (default false)`. When enabled, a `SparkListener` registered on the `SparkContext` listener bus handles `SparkListenerSQLExecutionEnd` events and proactively unregisters all shuffles written by the completed SQL query (or Dataset action), instead of waiting for the driver GC to trigger Spark's `ContextCleaner` or for the application to end. - New `CelebornSQLExecutionEndCleaner` listener collects shuffle ids from the query's final executed plan (AQE-aware) and cleans them up through the standard Spark cleanup entry point `shuffleDriverComponents.removeShuffle`, reusing the existing unregister pipeline with zero master/worker/protocol changes. - New `LifecycleManager.isAppShuffleRegistered` guard so only actually registered shuffles are unregistered. ### Why are the changes needed? In long-running Spark applications (`ThriftServer`, `Kyuubi`, `spark-sql`), one application runs many SQL queries, each producing new shuffles. Shuffle unregister is only triggered by Spark's GC-driven `ContextCleaner`, which may be delayed indefinitely in long sessions due to DAG/query-plan reference chains (see `CELEBORN-2244`), so obsolete shuffle data piles up on Celeborn worker disks for the whole session lifetime. ### Does this PR resolve a correctness bug? No. ### Does this PR introduce any user-facing change? Yes, a new client configuration `celeborn.client.spark.sql.queryEndShuffleCleanup.enabled (default false, opt-in)`. ### How was this patch tested? add ut `CelebornQueryEndShuffleCleanupSuite` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
