[
https://issues.apache.org/jira/browse/MAPREDUCE-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291984#comment-13291984
]
Shrinivas Joshi commented on MAPREDUCE-2374:
--------------------------------------------
We are noticing these "Text file busy" exceptions on an Hadoop 1.0.3
environment which are causing task failures with exit status 126. These
exceptions are non-deterministic. They occur when shExec.execute() call is made
in org.apache.hadoop.mapred.DefaultTaskController class. Return string from the
call to shExec.getOutput()is empty line. We captured the details of this
exception by logging the stack trace in the catch block.
Here is what the stack trace looks like:
org.apache.hadoop.util.Shell$ExitCodeException: bash:
/local4/1_0_3/hadoop-local/ttprivate/taskTracker/root/jobcache/job_201206081243_0001/attempt_201206081243_0001_m_000024_0/taskjvm.sh:
Text file busy
at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at
org.apache.hadoop.mapred.DefaultTaskController.launchTask(DefaultTaskController.java:131)
at
org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.runChild(JvmManager.java:498)
at
org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.run(JvmManager.java:472)
Also, all the log/work files corresponding to the failed tasks get cleaned up
and hence it becomes harder to debug the underlying cause.
Above patch has helped reduce these failures, however, they still occur. I am
not sure whether the "Text file busy" exception is a Hadoop issue or some other
issue though. I just thought sharing this information here would benefit
someone.
> Should not use PrintWriter to write taskjvm.sh
> ----------------------------------------------
>
> Key: MAPREDUCE-2374
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2374
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.22.1
>
> Attachments: mapreduce-2374-on-20sec.txt
>
>
> Our use of PrintWriter in TaskController.writeCommand is unsafe, since that
> class swallows all IO exceptions. We're not currently checking for errors,
> which I'm seeing result in occasional task failures with the message "Text
> file busy" - assumedly because the close() call is failing silently for some
> reason.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
