Aniket Bhatnagar created SPARK-18787:
----------------------------------------
Summary: spark.shuffle.io.preferDirectBufs does not completely
turn off direct buffer usage by Netty
Key: SPARK-18787
URL: https://issues.apache.org/jira/browse/SPARK-18787
Project: Spark
Issue Type: Bug
Affects Versions: 2.0.2
Reporter: Aniket Bhatnagar
The documentation for the configuration spark.shuffle.io.preferDirectBufs
states that it will force all allocations from Netty to be on-heap but this
currently does not happen. The reason is that preferDirect of Netty's
PooledByteBufAllocator doesn't completely eliminate use of off heap by Netty.
In order to completely stop netty from using off heap memory, we need to set
the following system properties:
- io.netty.noUnsafe=true
- io.netty.threadLocalDirectBufferSize=0
The proposal is to set properties (using System.setProperties) when the
executor starts (before any of the Netty classes load) or document these
properties to hint users on how to completely eliminate off Netty' heap
footprint.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
