L. C. Hsieh created SPARK-60603:
-----------------------------------
Summary: Shuffle service leaks canonical local dir cache entries
and canonicalizes the index path on every fetch
Key: SPARK-60603
URL: https://issues.apache.org/jira/browse/SPARK-60603
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
The block path verification added to ExternalShuffleBlockResolver
("[CORE][MINOR] Improve ESS file consistency",
spark.shuffle.service.verifyBlockPathsAtRead, default true) caches the
canonical form of every registered executor local dir in canonicalLocalDirs,
but never removes entries: not in applicationRemoved, not in executorRemoved.
On YARN every executor's local dirs are unique
(.../appcache/<appId>/blockmgr-<uuid>), so a long-running NodeManager-hosted
shuffle service grows this map without bound.
In addition, every shuffle block fetch canonicalizes both the index and the
data file path before the shuffleIndexCache lookup, although the index file is
only read on a cache miss. The new config is also not documented in
docs/configuration.md.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
