Ryan Blue created SPARK-16420:
---------------------------------
Summary: UnsafeShuffleWriter leaks compression streams with
off-heap memory.
Key: SPARK-16420
URL: https://issues.apache.org/jira/browse/SPARK-16420
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.0.0
Reporter: Ryan Blue
When merging spill files using Java file streams, {{UnsafeShuffleWriter}} opens
a decompression stream for each shuffle part and [never closes
them|https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java#L352-357].
When the compression codec holds off-heap resources, these aren't cleaned up
until the finalizer is called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
