Jasraj Dhanoa created SPARK-59907:
-------------------------------------
Summary: StreamingShuffleWriter leaks in-flight off-heap buffer
when write fails
Key: SPARK-59907
URL: https://issues.apache.org/jira/browse/SPARK-59907
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jasraj Dhanoa
When StreamingShuffleWriter.write() fails partway through, the in-flight
off-heap buffer is referenced only by a local variable, so task completion
cleanup does not release it. Because these buffers are not reclaimed by GC, the
executor loses that off-heap memory until it restarts.
Failure scenarios:
# The serializer throws while writing a record.
# Closing the serialization stream fails before a buffer is sent.
Expected behavior: when write() fails, StreamingShuffleWriter should release
the in-flight buffer and return its memory budget before rethrowing the
exception.
Reproduction (tested on 5.0.0-SNAPSHOT, master at commit 44929778453): run a
streaming shuffle write with a serializer that throws on a given record
(scenario 1) or on stream close (scenario 2). After the task fails, the
captured buffer still has refCnt 1 and one buffer's worth of memory budget is
not returned. With the fix, refCnt drops to 0 and the budget is fully returned.
Unit tests reproducing both scenarios will be included in the PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
