[
https://issues.apache.org/jira/browse/HDFS-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710939#comment-15710939
]
Brahma Reddy Battula commented on HDFS-11192:
---------------------------------------------
bq.Was it hitting its ulimit on # of threads/processes?
Yes, Some other process creating lot of threads which leads to cross user limit.
bq.A potential improvement will be to terminate NN if this happens.
To call terminate from code, there was no exception handling done for this part
of the code.
To terminate NN on OOM, java VM option {{-XX:OnOutOfMemoryError="kill -9 %p"}}
was passed, even that didnt take effect.
> OOM during Quota Initialization lead to Namenode hang
> -----------------------------------------------------
>
> Key: HDFS-11192
> URL: https://issues.apache.org/jira/browse/HDFS-11192
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Brahma Reddy Battula
> Assignee: Brahma Reddy Battula
> Attachments: namenodeThreadDump.out
>
>
> AFAIK ,In RecurisveTask Execution, When ForkjoinThreadpool's thread dies or
> not able to create,it will not notify the parent.Parent still waiting for the
> notify call..that's not timed waiting also.
> *Trace from Namenode log*
> {noformat}
> Exception in thread "ForkJoinPool-1-worker-2" Exception in thread
> "ForkJoinPool-1-worker-3" java.lang.OutOfMemoryError: unable to create new
> native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ForkJoinPool.createWorker(ForkJoinPool.java:1486)
> at
> java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1517)
> at
> java.util.concurrent.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1609)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:167)
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ForkJoinPool.createWorker(ForkJoinPool.java:1486)
> at
> java.util.concurrent.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1517)
> at
> java.util.concurrent.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1609)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:167)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
