[
https://issues.apache.org/jira/browse/ORC-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned ORC-409:
-----------------------------------------
> Changes for extending MemoryManagerImpl
> ---------------------------------------
>
> Key: ORC-409
> URL: https://issues.apache.org/jira/browse/ORC-409
> Project: ORC
> Issue Type: Bug
> Affects Versions: 1.5.4, 1.6.0
> Reporter: Prasanth Jayachandran
> Assignee: Prasanth Jayachandran
> Priority: Major
>
> Orc memory manager uses MemoryMX bean to get max heap usage and assumes 50%
> (configurable) is available for orc writers. This works well if container
> model where container size is close to Xmx. In LLAP for example, a single
> container runs multiple executors and a single task take up 50% of heap which
> is typically much more that the task executors own memory limit.
> Example: 128Gb container + 32 executors (assuming no cache) means there is
> 4GB available per executor. But a single task that opens multiple orc writers
> in the default case will assume 64GB memory is available for orc when in fact
> it should only use 2GB from memory per executor.
>
> WriterOption provides a mechanism to plug-in a custom memory manager for such
> scenarios which is greate. But if the custom memory manager want to extend
> MemoryManagerImpl class, MemoryManagerImpl should use getTotalMemoryPool()
> everywhere so that custom memory manager can just Override
> getTotalMemoryPool() method and rest of the code from MemoryManagerImpl.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
