Prasanth Jayachandran created ORC-409:
-----------------------------------------
Summary: Changes for extending MemoryManagerImpl
Key: ORC-409
URL: https://issues.apache.org/jira/browse/ORC-409
Project: ORC
Issue Type: Bug
Affects Versions: 1.5.4, 1.6.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
Orc memory manager uses MemoryMX bean to get max heap usage and assumes 50%
(configurable) is available for orc writers. This works well if container model
where container size is close to Xmx. In LLAP for example, a single container
runs multiple executors and a single task take up 50% of heap which is
typically much more that the task executors own memory limit.
Example: 128Gb container + 32 executors (assuming no cache) means there is 4GB
available per executor. But a single task that opens multiple orc writers in
the default case will assume 64GB memory is available for orc when in fact it
should only use 2GB from memory per executor.
WriterOption provides a mechanism to plug-in a custom memory manager for such
scenarios which is greate. But if the custom memory manager want to extend
MemoryManagerImpl class, MemoryManagerImpl should use getTotalMemoryPool()
everywhere so that custom memory manager can just Override getTotalMemoryPool()
method and rest of the code from MemoryManagerImpl.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
