HI,
I looked at log carefully, then I found: 2013-12-14 12:29:19,184 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_1386995306465_0001_02_000001 2013-12-14 12:29:19,185 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: FINISH_APPLICATION sent to absent application application_1386995306465_0001 It sounds like AMLauncher request container for AM, but node manager cannot get container. so how to solve it? On Sat, Dec 14, 2013 at 10:20 AM, Azuryy Yu <[email protected]> wrote: > Maybe My yarn was configured incorrect. > in the NM log: > 2013-12-14 10:11:48,660 DEBUG > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node's > health-status : true, > 2013-12-14 10:11:48,660 DEBUG > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: > test.slave152.com:8021 sending out status for 0 containers > 2013-12-14 10:11:48,687 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > Event EventType: KILL_CONTAINER sent to absent container > container_1386987056408_0001_02_000001 > 2013-12-14 10:11:48,688 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > Event EventType: FINISH_APPLICATION sent to absent application > application_1386987056408_0001 > > Does that mean there are 0 containers on this NM? > > > > On Sat, Dec 14, 2013 at 9:38 AM, Azuryy Yu <[email protected]> wrote: > >> Hi Vinod, >> I enabled DEBUG log on both RM and NM. >> RM Log: >> >> 2013-12-14 09:17:47,093 DEBUG org.apache.hadoop.security.SaslRpcClient: >> Sending sasl message state: NEGOTIATE >> >> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.security.SaslRpcClient: >> Received SASL message state: NEGOTIATE >> auths { >> method: "TOKEN" >> mechanism: "DIGEST-MD5" >> protocol: "" >> serverId: "default" >> challenge: >> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess" >> } >> >> 2013-12-14 09:17:47,216 DEBUG org.apache.hadoop.security.SaslRpcClient: >> Get token info proto:interface >> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB info:null >> 2013-12-14 09:17:47,216 ERROR >> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException >> as:appattempt_1386983771970_0001_000001 (auth:SIMPLE) >> cause:org.apache.hadoop.security.AccessControlException: Client cannot >> authenticate via:[TOKEN] >> >> NM Log: >> 2013-12-14 09:17:47,104 DEBUG >> org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hadoop >> (auth:SIMPLE) >> from:org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:150) >> 2013-12-14 09:17:47,111 DEBUG org.apache.hadoop.security.SaslRpcServer: >> Created SASL server with mechanism = DIGEST-MD5 >> 2013-12-14 09:17:47,205 DEBUG org.apache.hadoop.ipc.Server: Sending sasl >> message state: NEGOTIATE >> auths { >> method: "TOKEN" >> mechanism: "DIGEST-MD5" >> protocol: "" >> serverId: "default" >> challenge: >> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess" >> } >> >> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server >> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1 >> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server >> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1 >> Wrote 166 bytes. >> 2013-12-14 09:17:47,223 DEBUG org.apache.hadoop.ipc.Server: Socket Reader >> #1 for port 8021: disconnecting client 10.2.8.148:34175. Number of >> active connections: 0 >> >> I am using HA based on trunk, there is in the log as: >> 13/12/14 09:17:42 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = >> /home/hadoop/dn_socket >> 13/12/14 09:17:42 DEBUG hdfs.HAUtil: No HA service delegation token >> found for logical URI hdfs://test-cluster >> >> Please take a look, Thanks very much. >> >> >> >> On Sat, Dec 14, 2013 at 8:56 AM, Azuryy Yu <[email protected]> wrote: >> >>> Thanks Vinod, >>> >>> I am not chang MR AppMaster. I'll enable debug logging then paste here. >>> >>> >>> On Sat, Dec 14, 2013 at 2:36 AM, Vinod Kumar Vavilapalli < >>> [email protected]> wrote: >>> >>>> Is it stock MR or your modified MR AppMaster? >>>> >>>> Can you enable debug logging on NodeManager as well as the MR >>>> ApplicationMaster and trace what exactly is happening? We enabled TOKEN >>>> based auth by default, so you should look if AM has the NMToken or not. >>>> Also check the NM logs. >>>> >>>> Thanks, >>>> +Vinod >>>> >>>> On Dec 13, 2013, at 3:31 AM, Azuryy Yu <[email protected]> wrote: >>>> >>>> > Hi Dears, >>>> > >>>> > After YARN-961 committed to the trunk, we cannot sumbit MapReduce Job >>>> now. >>>> > >>>> > I am using trunk in the test unsecure cluster, when submit Job, it >>>> > complained: >>>> > 386931155525_0001_000002. Got exception: java.io.IOException: Failed >>>> on >>>> > local exception: java.io.IOException: >>>> > org.apache.hadoop.security.AccessControlException: Client cannot >>>> > authenticate via:[TOKEN]; Host Details : local host is: " >>>> > 10.2.8.148/10.2.8.148"; destination host is: "test.152.com":8021; >>>> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1410) >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1359) >>>> > at >>>> > >>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) >>>> > at com.sun.proxy.$Proxy23.startContainers(Unknown Source) >>>> > at >>>> > >>>> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96) >>>> > at >>>> > >>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118) >>>> > at >>>> > >>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249) >>>> > at >>>> > >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> > at >>>> > >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> > at java.lang.Thread.run(Thread.java:744) >>>> >>>> >>>> -- >>>> CONFIDENTIALITY NOTICE >>>> NOTICE: This message is intended for the use of the individual or >>>> entity to >>>> which it is addressed and may contain information that is confidential, >>>> privileged and exempt from disclosure under applicable law. If the >>>> reader >>>> of this message is not the intended recipient, you are hereby notified >>>> that >>>> any printing, copying, dissemination, distribution, disclosure or >>>> forwarding of this communication is strictly prohibited. If you have >>>> received this communication in error, please contact the sender >>>> immediately >>>> and delete it from your system. Thank You. >>>> >>> >>> >> >
