Resolved. It's my fault, I changed some code from the trunk, which leads to Token cannot generated.
On Sat, Dec 14, 2013 at 12:35 PM, Azuryy Yu <[email protected]> wrote: > HI, > > > I looked at log carefully, then I found: > 2013-12-14 12:29:19,184 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > Event EventType: KILL_CONTAINER sent to absent container > container_1386995306465_0001_02_000001 > 2013-12-14 12:29:19,185 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > Event EventType: FINISH_APPLICATION sent to absent application > application_1386995306465_0001 > > It sounds like AMLauncher request container for AM, but node manager > cannot get container. > > so how to solve it? > > > On Sat, Dec 14, 2013 at 10:20 AM, Azuryy Yu <[email protected]> wrote: > >> Maybe My yarn was configured incorrect. >> in the NM log: >> 2013-12-14 10:11:48,660 DEBUG >> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node's >> health-status : true, >> 2013-12-14 10:11:48,660 DEBUG >> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: >> test.slave152.com:8021 sending out status for 0 containers >> 2013-12-14 10:11:48,687 WARN >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: >> Event EventType: KILL_CONTAINER sent to absent container >> container_1386987056408_0001_02_000001 >> 2013-12-14 10:11:48,688 WARN >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: >> Event EventType: FINISH_APPLICATION sent to absent application >> application_1386987056408_0001 >> >> Does that mean there are 0 containers on this NM? >> >> >> >> On Sat, Dec 14, 2013 at 9:38 AM, Azuryy Yu <[email protected]> wrote: >> >>> Hi Vinod, >>> I enabled DEBUG log on both RM and NM. >>> RM Log: >>> >>> 2013-12-14 09:17:47,093 DEBUG org.apache.hadoop.security.SaslRpcClient: >>> Sending sasl message state: NEGOTIATE >>> >>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.security.SaslRpcClient: >>> Received SASL message state: NEGOTIATE >>> auths { >>> method: "TOKEN" >>> mechanism: "DIGEST-MD5" >>> protocol: "" >>> serverId: "default" >>> challenge: >>> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess" >>> } >>> >>> 2013-12-14 09:17:47,216 DEBUG org.apache.hadoop.security.SaslRpcClient: >>> Get token info proto:interface >>> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB info:null >>> 2013-12-14 09:17:47,216 ERROR >>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException >>> as:appattempt_1386983771970_0001_000001 (auth:SIMPLE) >>> cause:org.apache.hadoop.security.AccessControlException: Client cannot >>> authenticate via:[TOKEN] >>> >>> NM Log: >>> 2013-12-14 09:17:47,104 DEBUG >>> org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hadoop >>> (auth:SIMPLE) >>> from:org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:150) >>> 2013-12-14 09:17:47,111 DEBUG org.apache.hadoop.security.SaslRpcServer: >>> Created SASL server with mechanism = DIGEST-MD5 >>> 2013-12-14 09:17:47,205 DEBUG org.apache.hadoop.ipc.Server: Sending sasl >>> message state: NEGOTIATE >>> auths { >>> method: "TOKEN" >>> mechanism: "DIGEST-MD5" >>> protocol: "" >>> serverId: "default" >>> challenge: >>> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess" >>> } >>> >>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server >>> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1 >>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server >>> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1 >>> Wrote 166 bytes. >>> 2013-12-14 09:17:47,223 DEBUG org.apache.hadoop.ipc.Server: Socket >>> Reader #1 for port 8021: disconnecting client 10.2.8.148:34175. Number >>> of active connections: 0 >>> >>> I am using HA based on trunk, there is in the log as: >>> 13/12/14 09:17:42 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = >>> /home/hadoop/dn_socket >>> 13/12/14 09:17:42 DEBUG hdfs.HAUtil: No HA service delegation token >>> found for logical URI hdfs://test-cluster >>> >>> Please take a look, Thanks very much. >>> >>> >>> >>> On Sat, Dec 14, 2013 at 8:56 AM, Azuryy Yu <[email protected]> wrote: >>> >>>> Thanks Vinod, >>>> >>>> I am not chang MR AppMaster. I'll enable debug logging then paste >>>> here. >>>> >>>> >>>> On Sat, Dec 14, 2013 at 2:36 AM, Vinod Kumar Vavilapalli < >>>> [email protected]> wrote: >>>> >>>>> Is it stock MR or your modified MR AppMaster? >>>>> >>>>> Can you enable debug logging on NodeManager as well as the MR >>>>> ApplicationMaster and trace what exactly is happening? We enabled TOKEN >>>>> based auth by default, so you should look if AM has the NMToken or not. >>>>> Also check the NM logs. >>>>> >>>>> Thanks, >>>>> +Vinod >>>>> >>>>> On Dec 13, 2013, at 3:31 AM, Azuryy Yu <[email protected]> wrote: >>>>> >>>>> > Hi Dears, >>>>> > >>>>> > After YARN-961 committed to the trunk, we cannot sumbit MapReduce >>>>> Job now. >>>>> > >>>>> > I am using trunk in the test unsecure cluster, when submit Job, it >>>>> > complained: >>>>> > 386931155525_0001_000002. Got exception: java.io.IOException: Failed >>>>> on >>>>> > local exception: java.io.IOException: >>>>> > org.apache.hadoop.security.AccessControlException: Client cannot >>>>> > authenticate via:[TOKEN]; Host Details : local host is: " >>>>> > 10.2.8.148/10.2.8.148"; destination host is: "test.152.com":8021; >>>>> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) >>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1410) >>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1359) >>>>> > at >>>>> > >>>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) >>>>> > at com.sun.proxy.$Proxy23.startContainers(Unknown Source) >>>>> > at >>>>> > >>>>> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96) >>>>> > at >>>>> > >>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118) >>>>> > at >>>>> > >>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249) >>>>> > at >>>>> > >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> > at >>>>> > >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> > at java.lang.Thread.run(Thread.java:744) >>>>> >>>>> >>>>> -- >>>>> CONFIDENTIALITY NOTICE >>>>> NOTICE: This message is intended for the use of the individual or >>>>> entity to >>>>> which it is addressed and may contain information that is confidential, >>>>> privileged and exempt from disclosure under applicable law. If the >>>>> reader >>>>> of this message is not the intended recipient, you are hereby notified >>>>> that >>>>> any printing, copying, dissemination, distribution, disclosure or >>>>> forwarding of this communication is strictly prohibited. If you have >>>>> received this communication in error, please contact the sender >>>>> immediately >>>>> and delete it from your system. Thank You. >>>>> >>>> >>>> >>> >> >
