[
https://issues.apache.org/jira/browse/MAPREDUCE-5588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved MAPREDUCE-5588.
--------------------------------------
Resolution: Duplicate
Assignee: Chris Nauroth
Hello, [~moian].
There is a known bug in Jetty that can cause it to consume a ton of CPU and not
make any real progress. As a workaround, the Hadoop code monitors for this
condition and kills the tasktracker process if it finds Jetty consuming too
much CPU. (It's better to kill the tasktracker outright then leave it running
in an unresponsive state.) This monitoring thread was added in MAPREDUCE-3184.
If you review that issue, you can find more background information on
configuration and tuning.
The bug is set to be fixed in Jetty version 6.1.27, but we've been waiting a
long time on that release. We also can't easily jump to a new major version of
Jetty due to API backwards-compatibility issues. MAPREDUCE-2980 contains
ongoing discussion about our upgrade plan. As a workaround, there are also
patches available that you can apply on top of Jetty 6.1.26.
I'm going to resolve this as a duplicate of MAPREDUCE-2980.
> TaskTrackers get killed by JettyBugMonitor because of incredibly high cpu
> usage
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5588
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5588
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1.1.2
> Reporter: Anthony MOI
> Assignee: Chris Nauroth
> Labels: cpu-usage, jetty, tasktracker
>
> We are running a little cluster with 10 servers running task trackers. All of
> them are getting killed randomly with the following message
> {quote}
> 2013-10-17 11:32:31,037 FATAL org.apache.hadoop.mapred.JettyBugMonitor:
> ************************************************************
> Jetty CPU usage: 120093277.1%. This is greater than the fatal threshold
> mapred.tasktracker.jetty.cpu.threshold.fatal. Aborting JVM.
> ************************************************************
> 2013-10-17 11:32:31,039 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down TaskTracker
> ************************************************************/
> {quote}
> Everytime, the message notices a cpu usage above 120M%. Everything has been
> running for a while now (since 1.1.2 release) without any problems, and it
> started just like that.
> Any idea of what could cause this ?
--
This message was sent by Atlassian JIRA
(v6.1#6144)
