Ivan Sadikov created SPARK-62048:
------------------------------------
Summary: MemoryStore.putIteratorAsBytes leaks the compression
stream when unrolling a block fails
Key: SPARK-62048
URL: https://issues.apache.org/jira/browse/SPARK-62048
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.2.2, 5.0.0
Reporter: Ivan Sadikov
MemoryStore.putIteratorAsBytes creates a SerializedValuesHolder, whose
constructor opens a serialization stream wrapped by
SerializerManager.wrapForCompression. The stream is closed in only two cases:
- the block is fully unrolled: SerializedValuesHolder.getBuilder() closes it;
- the block is partially unrolled: the stream is handed to a
PartiallySerializedBlock, which closes it in discard() or
finishWritingToStream().
putIterator has no failure handling. If values.next() or
valuesHolder.storeValue() throws, the exception escapes putIteratorAsBytes and
nothing closes the stream or releases the unrolled buffers.
With the default zstd codec, ZStdCompressionCodec returns a
ZstdOutputStreamNoFinalizer. Its native compression context and buffers are
released only by close(), not by GC, so the executor leaks one native zstd
context each time a task fails while caching a block as serialized bytes (for
example MEMORY_ONLY_SER or MEMORY_AND_DISK_SER). Common non-fatal triggers:
- an exception thrown by the RDD's input iterator (user code);
- NotSerializableException from the Java serializer;
- KryoException from the Kryo serializer.
Because the task fails but the executor survives, the native memory adds up
across task retries and long-running executors and grows off-heap usage outside
the JVM heap limits.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]