For the first error, looked like one thread/process was writing the following shell file, /var/lib/hadoop/tmp/nm-local- dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh
but another thread/process was trying to execute it at the same time, which was why the error "bad interpreter: Text file busy" was thrown. Is there any possibility that 2.1.0-beta introduced some race condition here? I will look more into the second error. Thank, John On Sat, Aug 31, 2013 at 12:42 PM, Jian He <[email protected]> wrote: > Hi John > > This exception should indicate error from the container process. If the > container process exits with non-zero exit code, it will be logged. > In case of such errors, you'd better look at the per-container log see > what's happening there. > > Jian > > > On Fri, Aug 30, 2013 at 10:03 AM, Jian Fang <[email protected] > > wrote: > >> Hi, >> >> I upgraded to Hadoop 2.1.0-beta and suddenly I started to see error >> messages as follows. >> >> Exception from container-launch: >> org.apache.hadoop.util.Shell$ExitCodeException: bash: >> /var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1377823589199_0002/container_1377823589199_0002_01_000214/default_container_executor.sh: >> /bin/bash: bad interpreter: Text file busy >> >> at org.apache.hadoop.util.Shell.runCommand(Shell.java:458) >> at org.apache.hadoop.util.Shell.run(Shell.java:373) >> at >> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:578) >> at >> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:258) >> at >> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:74) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> at java.lang.Thread.run(Thread.java:662) >> >> >> >> cleanup failed for container container_1377823589199_0002_01_000214 : >> org.apache.hadoop.yarn.exceptions.YarnException: Container >> container_1377823589199_0002_01_000214 is not handled by this NodeManager >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> Method) >> at >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) >> at >> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:152) >> at >> org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106) >> at >> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:210) >> at >> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> at java.lang.Thread.run(Thread.java:662) >> >> Any thing wrong here? >> >> Thanks, >> >> John >> > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
