[
https://issues.apache.org/jira/browse/SPARK-26684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-26684:
------------------------------------
Assignee: Apache Spark
> Add logs when allocating large memory for PooledByteBufAllocator
> ----------------------------------------------------------------
>
> Key: SPARK-26684
> URL: https://issues.apache.org/jira/browse/SPARK-26684
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 2.4.0
> Reporter: liupengcheng
> Assignee: Apache Spark
> Priority: Trivial
>
> Currently, Spark use `PooledByteBufAllocator` to allocate memory for channel
> reading. However, the allocated heap/offheap memory size is not tracked.
> Sometimes, this make it difficult to find out the cause of OOM failures(for
> instance, direct memory oom). we have to dump the heap and use more advanced
> tools like MAT to locate the cause.
> Actually, we can add some logs for `PooledByteBufAllocator` when allocating
> large memory, which can facilitate the debugging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
