[
https://issues.apache.org/jira/browse/HIVE-10174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14392343#comment-14392343
]
Gopal V commented on HIVE-10174:
--------------------------------
That would be a good idea sometime down the line, because it would
differentiate between flushes due to the width of table vs multiple writers in
that case.
But the OOMs are due to an accident of allocation in most sparse & wide tables,
so I want to get HIVE-10036 in as a relatively benign fix for the real problem
there (suppressed streams etc).
For this particular bug, we need a Fragment memory manager to get past the
synchronization.
Since our operator pipeline is strictly single threaded for the
FileSinkOperator (even if we fork-join Filter/JOIN etc), we can consider a
thread-local as an equivalent Singleton instance, to handle the multi-spray
case.
> LLAP: ORC MemoryManager is singleton synchronized
> -------------------------------------------------
>
> Key: HIVE-10174
> URL: https://issues.apache.org/jira/browse/HIVE-10174
> Project: Hive
> Issue Type: Sub-task
> Components: File Formats
> Affects Versions: llap
> Reporter: Gopal V
> Attachments: orc-memorymanager-1.png, orc-memorymanager-2.png
>
>
> ORC MemoryManager::addedRow() checks are bad for LLAP multi-threaded
> performance.
> !orc-memorymanager-1.png!
> !orc-memorymanager-2.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
