[
https://issues.apache.org/jira/browse/SPARK-18787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-18787:
---------------------------------
Labels: bulk-closed (was: )
> spark.shuffle.io.preferDirectBufs does not completely turn off direct buffer
> usage by Netty
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-18787
> URL: https://issues.apache.org/jira/browse/SPARK-18787
> Project: Spark
> Issue Type: Bug
> Affects Versions: 2.0.2
> Reporter: Aniket Bhatnagar
> Priority: Major
> Labels: bulk-closed
>
> The documentation for the configuration spark.shuffle.io.preferDirectBufs
> states that it will force all allocations from Netty to be on-heap but this
> currently does not happen. The reason is that preferDirect parameter of
> Netty's PooledByteBufAllocator doesn't completely eliminate use of off heap
> by Netty. In order to completely stop netty from using off heap memory, we
> need to set the following system properties:
> - io.netty.noUnsafe=true
> - io.netty.threadLocalDirectBufferSize=0
> The proposal is to set properties (using System.setProperties) when the
> executor starts (before any of the Netty classes load) or document these
> properties to hint users on how to completely eliminate Netty' off heap
> footprint.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
