[
https://issues.apache.org/jira/browse/SPARK-26684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16764558#comment-16764558
]
Dongjoon Hyun commented on SPARK-26684:
---------------------------------------
Hi, [~liupengcheng].
As I commented on your PR, please contribute to `Netty` project itself. Inside
`Netty` project, it is one liner for each functions.
-
https://github.com/netty/netty/blob/4.1/buffer/src/main/java/io/netty/buffer/PooledByteBufAllocator.java#L321
> Add logs when allocating large memory for PooledByteBufAllocator
> ----------------------------------------------------------------
>
> Key: SPARK-26684
> URL: https://issues.apache.org/jira/browse/SPARK-26684
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Affects Versions: 2.4.0
> Reporter: liupengcheng
> Priority: Trivial
>
> Currently, Spark use `PooledByteBufAllocator` to allocate memory for channel
> reading. However, the allocated heap/offheap memory size is not tracked.
> Sometimes, this make it difficult to find out the cause of OOM failures(for
> instance, direct memory oom). we have to dump the heap and use more advanced
> tools like MAT to locate the cause.
> Actually, we can add some logs for `PooledByteBufAllocator` when allocating
> large memory, which can facilitate the debugging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
