Hi Harsh,
Please see if this is useful, I got a stack trace after the error has
occurred....
2013-08-06 00:55:30,559 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set
to
/tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004 =
file:/tmp/nm-local-dir/usercache/dsadm/appcache/application_1375716148174_0004
2013-08-06 00:55:31,017 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
exist: hdfs://isredeng/kishore/kk.ksh
2013-08-06 00:55:31,029 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
DEBUG: FAILED { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null }, File does
not exist: hdfs://isredeng/kishore/kk.ksh
2013-08-06 00:55:31,031 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource hdfs://isredeng/kishore/kk.ksh transitioned from DOWNLOADING to
FAILED
2013-08-06 00:55:31,034 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1375716148174_0004_01_000002 transitioned from
LOCALIZING to LOCALIZATION_FAILED
2013-08-06 00:55:31,035 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl:
Container container_1375716148174_0004_01_000002 sent RELEASE event on a
resource request { hdfs://isredeng/kishore/kk.ksh, 0, FILE, null } not
present in cache.
2013-08-06 00:55:31,036 WARN org.apache.hadoop.ipc.Client: interrupted
waiting to send rpc request to server
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1290)
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:229)
at java.util.concurrent.FutureTask.get(FutureTask.java:94)
at
org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:930)
at org.apache.hadoop.ipc.Client.call(Client.java:1285)
at org.apache.hadoop.ipc.Client.call(Client.java:1264)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at $Proxy22.heartbeat(Unknown Source)
at
org.apache.hadoop.yarn.server.nodemanager.api.impl.pb.client.LocalizationProtocolPBClientImpl.heartbeat(LocalizationProtocolPBClientImpl.java:62)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.localizeFiles(ContainerLocalizer.java:249)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:163)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:106)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:979)
And here is my code snippet:
ContainerLaunchContext ctx =
Records.newRecord(ContainerLaunchContext.class);
ctx.setEnvironment(oshEnv);
// Set the local resources
Map<String, LocalResource> localResources = new HashMap<String,
LocalResource>();
LocalResource shellRsrc = Records.newRecord(LocalResource.class);
shellRsrc.setType(LocalResourceType.FILE);
shellRsrc.setVisibility(LocalResourceVisibility.APPLICATION);
String shellScriptPath = "hdfs://isredeng//kishore/kk.ksh";
try {
shellRsrc.setResource(ConverterUtils.getYarnUrlFromURI(new
URI(shellScriptPath)));
} catch (URISyntaxException e) {
LOG.error("Error when trying to use shell script path specified"
+ " in env, path=" + shellScriptPath);
e.printStackTrace();
}
shellRsrc.setTimestamp(0/*shellScriptPathTimestamp*/);
shellRsrc.setSize(0/*shellScriptPathLen*/);
String ExecShellStringPath = "ExecShellScript.sh";
localResources.put(ExecShellStringPath, shellRsrc);
ctx.setLocalResources(localResources);
Please let me know if you need anything else.
Thanks,
Kishore
On Tue, Aug 6, 2013 at 12:05 AM, Harsh J <ha...@cloudera.com> wrote:
> The detail is insufficient to answer why. You should also have gotten
> a trace after it, can you post that? If possible, also the relevant
> snippets of code.
>
> On Mon, Aug 5, 2013 at 6:36 PM, Krishna Kishore Bonagiri
> <write2kish...@gmail.com> wrote:
> > Hi Harsh,
> > Thanks for the quick and detailed reply, it really helps. I am trying to
> > use it and getting this error in node manager's log:
> >
> > 2013-08-05 08:57:28,867 ERROR
> > org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException
> > as:dsadm (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not
> > exist: hdfs://isredeng/kishore/kk.ksh
> >
> >
> > This file is there on the machine with name "isredeng", I could do ls for
> > that file as below:
> >
> > -bash-4.1$ hadoop fs -ls kishore/kk.ksh
> > 13/08/05 09:01:03 WARN util.NativeCodeLoader: Unable to load
> native-hadoop
> > library for your platform... using builtin-java classes where applicable
> > Found 1 items
> > -rw-r--r-- 3 dsadm supergroup 1046 2013-08-05 08:48
> kishore/kk.ksh
> >
> > Note: I am using a single node cluster
> >
> > Thanks,
> > Kishore
> >
> >
> >
> >
> > On Mon, Aug 5, 2013 at 3:00 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> The string for each LocalResource in the map can be anything that
> >> serves as a common identifier name for your application. At execution
> >> time, the passed resource filename will be aliased to the name you've
> >> mapped it to, so that the application code need not track special
> >> names. The behavior is very similar to how you can, in MR, define a
> >> symlink name for a DistributedCache entry (e.g. foo.jar#bar.jar).
> >>
> >> For an example, checkout the DistributedShell app sources.
> >>
> >> Over [1], you can see we take a user provided file path to a shell
> >> script. This can be named anything as it is user-supplied.
> >> Onto [2], we define this as a local resource [2.1] and embed it with a
> >> different name (the string you ask about) [2.2], as defined at [3] as
> >> an application reference-able constant.
> >> Note that in [4], we add to the Container arguments the aliased name
> >> we mapped it to (i.e. [3]) and not the original filename we received
> >> from the user. The resource is placed on the container with this name
> >> instead, so thats what we choose to execute.
> >>
> >> [1] -
> >>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L390
> >>
> >> [2] - [2.1]
> >>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L764
> >> and [2.2]
> >>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L780
> >>
> >> [3] -
> >>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L205
> >>
> >> [4] -
> >>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java#L791
> >>
> >> On Mon, Aug 5, 2013 at 2:44 PM, Krishna Kishore Bonagiri
> >> <write2kish...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > Can someone please tell me what is the use of calling
> >> > setLocalResources()
> >> > on ContainerLaunchContext?
> >> >
> >> > And, also an example of how to use this will help...
> >> >
> >> > I couldn't guess what is the String in the map that is passed to
> >> > setLocalResources() like below:
> >> >
> >> > // Set the local resources
> >> > Map<String, LocalResource> localResources = new HashMap<String,
> >> > LocalResource>();
> >> >
> >> > Thanks,
> >> > Kishore
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>
