[jira] [Commented] (HIVE-10174) LLAP: ORC MemoryManager is singleton synchronized
[ https://issues.apache.org/jira/browse/HIVE-10174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392343#comment-14392343 ] Gopal V commented on HIVE-10174: That would be a good idea sometime down the line, because it would differentiate between flushes due to the width of table vs multiple writers in that case. But the OOMs are due to an accident of allocation in most sparse wide tables, so I want to get HIVE-10036 in as a relatively benign fix for the real problem there (suppressed streams etc). For this particular bug, we need a Fragment memory manager to get past the synchronization. Since our operator pipeline is strictly single threaded for the FileSinkOperator (even if we fork-join Filter/JOIN etc), we can consider a thread-local as an equivalent Singleton instance, to handle the multi-spray case. LLAP: ORC MemoryManager is singleton synchronized - Key: HIVE-10174 URL: https://issues.apache.org/jira/browse/HIVE-10174 Project: Hive Issue Type: Sub-task Components: File Formats Affects Versions: llap Reporter: Gopal V Attachments: orc-memorymanager-1.png, orc-memorymanager-2.png ORC MemoryManager::addedRow() checks are bad for LLAP multi-threaded performance. !orc-memorymanager-1.png! !orc-memorymanager-2.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10174) LLAP: ORC MemoryManager is singleton synchronized
[ https://issues.apache.org/jira/browse/HIVE-10174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392337#comment-14392337 ] Prasanth Jayachandran commented on HIVE-10174: -- Should we have single writer memory manager (dynamic partition with sort optimization or unpartitioned table) and multi writer memory manager (dynamic partition with sort optimization disabled)? LLAP: ORC MemoryManager is singleton synchronized - Key: HIVE-10174 URL: https://issues.apache.org/jira/browse/HIVE-10174 Project: Hive Issue Type: Sub-task Components: File Formats Affects Versions: llap Reporter: Gopal V Attachments: orc-memorymanager-1.png, orc-memorymanager-2.png ORC MemoryManager::addedRow() checks are bad for LLAP multi-threaded performance. !orc-memorymanager-1.png! !orc-memorymanager-2.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10174) LLAP: ORC MemoryManager is singleton synchronized
[ https://issues.apache.org/jira/browse/HIVE-10174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389958#comment-14389958 ] Gopal V commented on HIVE-10174: Performance difference is somewhere along the lines of 34s with the MemoryManager + addRow synchronized blocks vs 9s without the MemoryManager and the addRow synchronized(this). To be looked at when we're writing ORC out of LLAP. LLAP: ORC MemoryManager is singleton synchronized - Key: HIVE-10174 URL: https://issues.apache.org/jira/browse/HIVE-10174 Project: Hive Issue Type: Sub-task Components: File Formats Affects Versions: llap Reporter: Gopal V Attachments: orc-memorymanager-1.png, orc-memorymanager-2.png ORC MemoryManager::addedRow() checks are bad for LLAP multi-threaded performance. !orc-memorymanager-1.png! !orc-memorymanager-2.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
