[
https://issues.apache.org/jira/browse/SPARK-18787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733902#comment-15733902
]
Sean Owen commented on SPARK-18787:
-----------------------------------
CC [~zsxwing]
The tricky thing is that I think these classes may load before we know whether
this config is enabled?
> spark.shuffle.io.preferDirectBufs does not completely turn off direct buffer
> usage by Netty
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-18787
> URL: https://issues.apache.org/jira/browse/SPARK-18787
> Project: Spark
> Issue Type: Bug
> Affects Versions: 2.0.2
> Reporter: Aniket Bhatnagar
>
> The documentation for the configuration spark.shuffle.io.preferDirectBufs
> states that it will force all allocations from Netty to be on-heap but this
> currently does not happen. The reason is that preferDirect of Netty's
> PooledByteBufAllocator doesn't completely eliminate use of off heap by Netty.
> In order to completely stop netty from using off heap memory, we need to set
> the following system properties:
> - io.netty.noUnsafe=true
> - io.netty.threadLocalDirectBufferSize=0
> The proposal is to set properties (using System.setProperties) when the
> executor starts (before any of the Netty classes load) or document these
> properties to hint users on how to completely eliminate off Netty' heap
> footprint.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
