Saurabh Damle created SPARK-59776:
-------------------------------------
Summary: Dynamic allocation shuffle tracking never releases
executors when SQL shuffle file cleanup is enabled
Key: SPARK-59776
URL: https://issues.apache.org/jira/browse/SPARK-59776
Project: Spark
Issue Type: Bug
Components: Spark Core, SQL
Affects Versions: 4.3.0
Reporter: Saurabh Damle
When `spark.sql.classic.shuffleDependency.fileCleanup.enabled` (or the
connect/thriftserver
variants) is enabled, SQLExecution.cleanupShuffleDependencies removes shuffle
files by calling
`sc.shuffleDriverComponents.removeShuffle` directly. This bypasses
ContextCleaner, so
`CleanerListener.shuffleCleaned` is never invoked and ExecutorMonitor continues
to consider
those shuffles active.
With `spark.dynamicAllocation.shuffleTracking.enabled=true` and the default
(infinite)
`spark.dynamicAllocation.shuffleTracking.timeout`, executors that hold the
(already deleted)
shuffle data are never released after actions like collect() / toPandas().
Proposed fix: notify ContextCleaner listeners when SQL cleanup removes shuffle
files, so
ExecutorMonitor drops the shuffle IDs.
Reported in https://github.com/apache/spark/issues/59014
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
