Hi there, I have an Application that is trying to launch an AM , but the localization is failing with the below error message. The Resource visiblity is set to private which means the localizer will run through a container executor as a user that submit the job. I checked that hdfs dfs -ls / command is working fine as a user sumbitting the Application to make sure kerberos credentials for this user is not to be blamed. a mapreduce pi example could run successfully. Any idea what could cause this kind of issue? Thanks Reyane OUKPEDJO
NODE MANAGER LOGS HERE 2015-04-27 22:30:48,682 INFO authorize.ServiceAuthorizationManager (ServiceAuthorizationManager.java:authorize(114)) - Authorization successful for testing (auth:TOKEN) for protocol=interface org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB2015-04-27 22:30:49,463 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:update(932)) - DEBUG: FAILED { hdfs://datanode5.in.ibm.com:8020/user/dsadm/.staging/AppMaster.jar, 1430154129414, FILE, null }, Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "datanode3.in.ibm.com/9.126.90.234"; destination host is: "datanode5.in.ibm.com":8020;2015-04-27 22:30:49,463 INFO localizer.LocalizedResource (LocalizedResource.java:handle(196)) - Resource hdfs://datanode5.in.ibm.com:8020/user/dsadm/.staging/AppMaster.jar transitioned from DOWNLOADING to FAILED2015-04-27 22:30:49,463 INFO container.Container (ContainerImpl.java:handle(901)) - Container container_1429925144518_0016_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED2015-04-27 22:30:49,464 INFO localizer.LocalResourcesTrackerImpl (LocalResourcesTrackerImpl.java:handle(137)) - Container container_1429925144518_0016_01_000001 sent RELEASE event on a resource request { hdfs://datanode5.in.ibm.com:8020/user/dsadm/.staging/AppMaster.jar, 1430154129414, FILE, null } not present in cache.2015-04-27 22:30:49,464 WARN nodemanager.NMAuditLogger (NMAuditLogger.java:logFailure(150)) - USER=dsadm OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1429925144518_0016 CONTAINERID=container_1429925144518_0016_01_0000012015-04-27 22:30:49,464 INFO container.Container (ContainerImpl.java:handle(901)) - Container container_1429925144518_0016_01_000001 transitioned from LOCALIZATION_FAILED to DONE2015-04-27 22:30:49,465 INFO application.Application (ApplicationImpl.java:transition(339)) - Removing container_1429925144518_0016_01_000001 from application application_1429925144518_00162015-04-27 22:30:49,465 INFO monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:isEnabled(169)) - Neither virutal-memory nor physical-memory monitoring is needed. Not running the monitor-thread2015-04-27 22:30:49,465 INFO containermanager.AuxServices (AuxServices.java:handle(175)) - Got event CONTAINER_STOP for appId application_1429925144518_0016@