JacobZheng created SPARK-47910:
----------------------------------
Summary: Memory leak when interrupting shuffle write using zstd
compression
Key: SPARK-47910
URL: https://issues.apache.org/jira/browse/SPARK-47910
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.5.0, 3.4.0, 3.3.0, 3.2.0
Reporter: JacobZheng
When spark.sql.execution.interruptOnCancel=true and
spark.io.compression.codec=zstd, a memory leak was found when tasks were
cancelled at specific times. The reason for this is that cancelling a task
interrupts the shuffle write, which then calls
org.apache.spark.storage.DiskBlockObjectWriter#closeResources. this process
then only closes the ManualCloseOutputStream, which is wrapped with this
ZstdInputStreamNoFinalizer will not be closed. Moreover,
ZstdInputStreamNoFinalizer doesn't implement Finalizer so it won't be reclaimed
by GC automatically.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
