Lennon Chin created SPARK-56252:
-----------------------------------
Summary: History server disk store lease should remove temp path
delete hook after commit and rollback to avoid memory leak
Key: SPARK-56252
URL: https://issues.apache.org/jira/browse/SPARK-56252
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.1.1
Reporter: Lennon Chin
In the History Server DiskManager, when acquiring a lease, a temporary
directory is created on the disk, and a JVM shutdown hook is registered used to
remove this temporary directory. When the lease is committed or rolled back,
the temporary directory is either renamed to the target directory or deleted,
but the registered JVM shutdown hook remains in the ShutdownHookManager,
causing a memory leak. Additionally, a large number of residual hooks will
execute when the History Server restarts, severely slowing down the restart
process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
