[
https://issues.apache.org/jira/browse/ORC-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631233#comment-16631233
]
ASF GitHub Bot commented on ORC-409:
------------------------------------
GitHub user prasanthj opened a pull request:
https://github.com/apache/orc/pull/313
ORC-409: Changes for extending MemoryManagerImpl
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/prasanthj/orc ORC-409
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/orc/pull/313.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #313
----
commit 9cc833d560da188fd48ece42c31d1daa9ea6bc08
Author: Prasanth Jayachandran <prasanthj@...>
Date: 2018-09-28T00:51:57Z
ORC-409: Changes for extending MemoryManagerImpl
----
> Changes for extending MemoryManagerImpl
> ---------------------------------------
>
> Key: ORC-409
> URL: https://issues.apache.org/jira/browse/ORC-409
> Project: ORC
> Issue Type: Bug
> Affects Versions: 1.5.4, 1.6.0
> Reporter: Prasanth Jayachandran
> Assignee: Prasanth Jayachandran
> Priority: Major
>
> Orc memory manager uses MemoryMX bean to get max heap usage and assumes 50%
> (configurable) is available for orc writers. This works well if container
> model where container size is close to Xmx. In LLAP for example, a single
> container runs multiple executors and a single task take up 50% of heap which
> is typically much more that the task executors own memory limit.
> Example: 128Gb container + 32 executors (assuming no cache) means there is
> 4GB available per executor. But a single task that opens multiple orc writers
> in the default case will assume 64GB memory is available for orc when in fact
> it should only use 2GB from memory per executor.
>
> WriterOption provides a mechanism to plug-in a custom memory manager for such
> scenarios which is greate. But if the custom memory manager want to extend
> MemoryManagerImpl class, MemoryManagerImpl should use getTotalMemoryPool()
> everywhere so that custom memory manager can just Override
> getTotalMemoryPool() method and rest of the code from MemoryManagerImpl.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
