GitHub user aarondav opened a pull request:
https://github.com/apache/spark/pull/3465
[SPARK-4516] Avoid allocating unnecessarily Netty PooledByteBufAllocators
Turns out we are allocating an allocator pool for every TransportClient
(which means that the number increases with the number of nodes in the
cluster), when really we should just reuse one for all clients.
This patch, as expected, greatly decreases off-heap memory allocation, and
appears to make allocation only proportional to the number of cores.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aarondav/spark fewer-pools
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3465.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3465
----
commit 36c49da5bbc503ddadce8b3b88e5748a5f4163c9
Author: Aaron Davidson <[email protected]>
Date: 2014-11-26T02:15:56Z
[SPARK-4516] Avoid allocating unnecessarily Netty PooledByteBufAllocators
Turns out we are allocating an allocator pool for every TransportClient
(which means that the number increases with the number of nodes in the
cluster), when really we should just reuse one for all clients.
This patch, as expected, massively decreases off-heap memory allocation,
and appears to make allocation only proportional to the number of cores.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
