Hi there,
I have an Application that is trying to launch an AM , but the localization is
failing with the below error message. The Resource visiblity is set to private
which means the localizer will run through a container executor as a user that
submit the job. I checked that hdfs dfs -ls / command is working fine as a user
sumbitting the Application to make sure kerberos credentials for this user is
not to be blamed. a mapreduce pi example could run successfully. Any idea
what could cause this kind of issue?
Thanks
Reyane OUKPEDJO
NODE MANAGER LOGS HERE 2015-04-27 22:30:48,682 INFO
authorize.ServiceAuthorizationManager
(ServiceAuthorizationManager.java:authorize(114)) - Authorization successful
for testing (auth:TOKEN) for protocol=interface
org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB2015-04-27
22:30:49,463 INFO localizer.ResourceLocalizationService
(ResourceLocalizationService.java:update(932)) - DEBUG: FAILED {
hdfs://datanode5.in.ibm.com:8020/user/dsadm/.staging/AppMaster.jar,
1430154129414, FILE, null }, Failed on local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]; Host Details : local host is:
"datanode3.in.ibm.com/9.126.90.234"; destination host is:
"datanode5.in.ibm.com":8020;2015-04-27 22:30:49,463 INFO
localizer.LocalizedResource (LocalizedResource.java:handle(196)) - Resource
hdfs://datanode5.in.ibm.com:8020/user/dsadm/.staging/AppMaster.jar transitioned
from DOWNLOADING to FAILED2015-04-27 22:30:49,463 INFO container.Container
(ContainerImpl.java:handle(901)) - Container
container_1429925144518_0016_01_000001 transitioned from LOCALIZING to
LOCALIZATION_FAILED2015-04-27 22:30:49,464 INFO
localizer.LocalResourcesTrackerImpl
(LocalResourcesTrackerImpl.java:handle(137)) - Container
container_1429925144518_0016_01_000001 sent RELEASE event on a resource request
{ hdfs://datanode5.in.ibm.com:8020/user/dsadm/.staging/AppMaster.jar,
1430154129414, FILE, null } not present in cache.2015-04-27 22:30:49,464 WARN
nodemanager.NMAuditLogger (NMAuditLogger.java:logFailure(150)) - USER=dsadm
OPERATION=Container Finished - Failed TARGET=ContainerImpl
RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED
APPID=application_1429925144518_0016
CONTAINERID=container_1429925144518_0016_01_0000012015-04-27 22:30:49,464 INFO
container.Container (ContainerImpl.java:handle(901)) - Container
container_1429925144518_0016_01_000001 transitioned from LOCALIZATION_FAILED to
DONE2015-04-27 22:30:49,465 INFO application.Application
(ApplicationImpl.java:transition(339)) - Removing
container_1429925144518_0016_01_000001 from application
application_1429925144518_00162015-04-27 22:30:49,465 INFO
monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:isEnabled(169)) -
Neither virutal-memory nor physical-memory monitoring is needed. Not running
the monitor-thread2015-04-27 22:30:49,465 INFO containermanager.AuxServices
(AuxServices.java:handle(175)) - Got event CONTAINER_STOP for appId
application_1429925144518_0016@
