[
https://issues.apache.org/jira/browse/SPARK-3018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin updated SPARK-3018:
-------------------------------
Description: BlockFetcherIterator retains ReferenceCountedBuffers returned
by client.fetchBlocks. Those buffers are released when the iterators are
traversed fully. In the case of task failures or completion without exhausting
the iterator, this could lead to memory leak. (was: BlockFetcherIterator
retains ReferenceCountedBuffers returned by client.fetchBlocks. Those buffers
are released when the iterators are traversed fully. In the case of task
failures or completion without depleting the iterator, this could lead to
memory leak.)
> Release all BlockFetcherIterator upon task completion/failure
> -------------------------------------------------------------
>
> Key: SPARK-3018
> URL: https://issues.apache.org/jira/browse/SPARK-3018
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle, Spark Core
> Reporter: Reynold Xin
>
> BlockFetcherIterator retains ReferenceCountedBuffers returned by
> client.fetchBlocks. Those buffers are released when the iterators are
> traversed fully. In the case of task failures or completion without
> exhausting the iterator, this could lead to memory leak.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
