[
https://issues.apache.org/jira/browse/HDFS-13393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665718#comment-16665718
]
Wei-Chiu Chuang commented on HDFS-13393:
----------------------------------------
Sample logs:
{quote}
2017-12-19 03:38:14,786 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
dn1.example.com:50010:DataXceiver error processing WRITE_BLOCK operation src:
/10.112.197.22:46888 dst: /10.112.197.21:50010
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
{quote}
{quote}
17/11/30 16:17:29 WARN balancer.Dispatcher: Dispatcher thread failed
java.lang.OutOfMemoryError: unable to create new native thread
{quote}
> Improve OOM logging
> -------------------
>
> Key: HDFS-13393
> URL: https://issues.apache.org/jira/browse/HDFS-13393
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, datanode
> Reporter: Wei-Chiu Chuang
> Assignee: Gabor Bota
> Priority: Major
>
> It is not uncommon to find "java.lang.OutOfMemoryError: unable to create new
> native thread" errors in a HDFS cluster. Most often this happens when
> DataNode creating DataXceiver threads, or when balancer creates threads for
> moving blocks around.
> In most of cases, the "OOM" is a symptom of number of threads reaching system
> limit, rather than actually running out of memory, and the current logging of
> this message is usually misleading (suggesting this is due to insufficient
> memory)
> How about capturing the OOM, and if it is due to "unable to create new native
> thread", print some more helpful message like "bump your ulimit" or "take a
> jstack of the process"?
> Even better, surface this error to make it more visible. It usually takes a
> while for an in-depth investigation after users notice some job fails, by the
> time the evidences may already been gone (like jstack output).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
