[
https://issues.apache.org/jira/browse/MAPREDUCE-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754954#comment-16754954
]
lujie commented on MAPREDUCE-7178:
----------------------------------
Seem many casees can cause this bug:
{code:java}
2019-01-29 15:11:04,688 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : ExitCodeException exitCode=143:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008)
at org.apache.hadoop.util.Shell.run(Shell.java:901)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1307)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1289)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:865)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:252)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:232)
at
org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:331)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:320)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:351)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:401)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:464)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:987)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:975)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:652)
at org.apache.hadoop.mapred.YarnChild.writeLocalJobFile(YarnChild.java:373)
at org.apache.hadoop.mapred.YarnChild.configureTask(YarnChild.java:355)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:150)
2019-01-29 15:11:04,694 INFO [main] org.apache.hadoop.mapred.Task: Running
cleanup for the task
2019-01-29 15:11:04,694 INFO [main] org.apache.hadoop.mapred.YarnChild:
Exception cleaning up:
java.lang.NullPointerException
at org.apache.hadoop.mapred.Task.taskCleanup(Task.java:1458)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:194)
{code}
> NPE while YarnChild shudown
> ---------------------------
>
> Key: MAPREDUCE-7178
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7178
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: lujie
> Assignee: lujie
> Priority: Major
> Attachments: MR-7178_1.patch, yarnchild.log
>
>
> In YarnChild.main
> {code:java}
> try{
> logSyncer = TaskLog.createLogSyncer();//line 168
> ....
> taskFinal.run(job, umbilical); //line 178
> }catch (Exception exception) {//line 187
> LOG.warn("Exception running child : "
> + StringUtils.stringifyException(exception));
> .................
> task.taskCleanup(umbilical);// line 200
> }{code}
> At line 178. it will initialize the task.committer, but the line168 may throw
> exception, it will skip initialize the task.committer, hence task.committer
> == null. Line 187 will catch this exception and do clean up(line 200), code
> line 200 will use task.committer without null check, hence NPE happens
> {code:java}
> 2019-01-23 16:59:42,864 INFO [main] org.apache.hadoop.mapred.YarnChild:
> Exception cleaning up: java.lang.NullPointerException
> at org.apache.hadoop.mapred.Task.taskCleanup(Task.java:1458)
> at org.apache.hadoop.mapred.YarnChild$3.run(YarnChild.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:197)
> {code}
> So why line168 may throw exception, below log give a example:
> {code:java}
> 2019-01-23 16:59:42,857 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.IllegalStateException: Shutdown in
> progress, cannot add a shutdownHook
> at
> org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:299)
> at org.apache.hadoop.mapred.TaskLog.createLogSyncer(TaskLog.java:340)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
