[
https://issues.apache.org/jira/browse/HIVE-15693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839399#comment-15839399
]
Siddharth Seth commented on HIVE-15693:
---------------------------------------
Instead of 2x executors - think this needs to be based on the concurrency. New
config parameter to set an upper bound? Lower bound to number of executors?
Killed attempts is more likely to be based on number of AMs communicating,
rather than the number of executors in the daemon.
Eventually, I think we need to have a certain number of threads per AM - and
also ensure that all threads don't end up blocking because of one bad AM. I'll
create a follow up jira for this.
> LLAP: cached threadpool in AMReporter creates too many threads leading to OOM
> -----------------------------------------------------------------------------
>
> Key: HIVE-15693
> URL: https://issues.apache.org/jira/browse/HIVE-15693
> Project: Hive
> Issue Type: Bug
> Components: llap
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Critical
> Attachments: HIVE-15693.1.patch
>
>
> branch: master
> {noformat}
> 2017-01-22T19:52:42,470 WARN [IPC Server handler 3 on 34642 ()]
> org.apache.hadoop.ipc.Server: IPC Server handler 3 on 34642, call
> org.apache.hadoop.hive.llap.protocol.LlapProtocolBlockingPB.submitWork
> ...Call#17257 Retry#0
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method) ~[?:1.8.0_77]
> at java.lang.Thread.start(Thread.java:714) [?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
> ~[?:1.8.0_77]
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
> ~[?:1.8.0_77]
> at
> com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:480)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:61)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.daemon.impl.AMReporter.taskKilled(AMReporter.java:231)
> ~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl$KilledTaskHandlerImpl.taskKilled(ContainerRunnerImpl.java:501)
> ~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
