Maybe My yarn was configured incorrect. in the NM log: 2013-12-14 10:11:48,660 DEBUG org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node's health-status : true, 2013-12-14 10:11:48,660 DEBUG org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: test.slave152.com:8021 sending out status for 0 containers 2013-12-14 10:11:48,687 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: KILL_CONTAINER sent to absent container container_1386987056408_0001_02_000001 2013-12-14 10:11:48,688 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Event EventType: FINISH_APPLICATION sent to absent application application_1386987056408_0001
Does that mean there are 0 containers on this NM? On Sat, Dec 14, 2013 at 9:38 AM, Azuryy Yu <[email protected]> wrote: > Hi Vinod, > I enabled DEBUG log on both RM and NM. > RM Log: > > 2013-12-14 09:17:47,093 DEBUG org.apache.hadoop.security.SaslRpcClient: > Sending sasl message state: NEGOTIATE > > 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.security.SaslRpcClient: > Received SASL message state: NEGOTIATE > auths { > method: "TOKEN" > mechanism: "DIGEST-MD5" > protocol: "" > serverId: "default" > challenge: > "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess" > } > > 2013-12-14 09:17:47,216 DEBUG org.apache.hadoop.security.SaslRpcClient: > Get token info proto:interface > org.apache.hadoop.yarn.api.ContainerManagementProtocolPB info:null > 2013-12-14 09:17:47,216 ERROR > org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException > as:appattempt_1386983771970_0001_000001 (auth:SIMPLE) > cause:org.apache.hadoop.security.AccessControlException: Client cannot > authenticate via:[TOKEN] > > NM Log: > 2013-12-14 09:17:47,104 DEBUG > org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hadoop > (auth:SIMPLE) > from:org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:150) > 2013-12-14 09:17:47,111 DEBUG org.apache.hadoop.security.SaslRpcServer: > Created SASL server with mechanism = DIGEST-MD5 > 2013-12-14 09:17:47,205 DEBUG org.apache.hadoop.ipc.Server: Sending sasl > message state: NEGOTIATE > auths { > method: "TOKEN" > mechanism: "DIGEST-MD5" > protocol: "" > serverId: "default" > challenge: > "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess" > } > > 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server > Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1 > 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server > Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1 > Wrote 166 bytes. > 2013-12-14 09:17:47,223 DEBUG org.apache.hadoop.ipc.Server: Socket Reader > #1 for port 8021: disconnecting client 10.2.8.148:34175. Number of active > connections: 0 > > I am using HA based on trunk, there is in the log as: > 13/12/14 09:17:42 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = > /home/hadoop/dn_socket > 13/12/14 09:17:42 DEBUG hdfs.HAUtil: No HA service delegation token found > for logical URI hdfs://test-cluster > > Please take a look, Thanks very much. > > > > On Sat, Dec 14, 2013 at 8:56 AM, Azuryy Yu <[email protected]> wrote: > >> Thanks Vinod, >> >> I am not chang MR AppMaster. I'll enable debug logging then paste here. >> >> >> On Sat, Dec 14, 2013 at 2:36 AM, Vinod Kumar Vavilapalli < >> [email protected]> wrote: >> >>> Is it stock MR or your modified MR AppMaster? >>> >>> Can you enable debug logging on NodeManager as well as the MR >>> ApplicationMaster and trace what exactly is happening? We enabled TOKEN >>> based auth by default, so you should look if AM has the NMToken or not. >>> Also check the NM logs. >>> >>> Thanks, >>> +Vinod >>> >>> On Dec 13, 2013, at 3:31 AM, Azuryy Yu <[email protected]> wrote: >>> >>> > Hi Dears, >>> > >>> > After YARN-961 committed to the trunk, we cannot sumbit MapReduce Job >>> now. >>> > >>> > I am using trunk in the test unsecure cluster, when submit Job, it >>> > complained: >>> > 386931155525_0001_000002. Got exception: java.io.IOException: Failed on >>> > local exception: java.io.IOException: >>> > org.apache.hadoop.security.AccessControlException: Client cannot >>> > authenticate via:[TOKEN]; Host Details : local host is: " >>> > 10.2.8.148/10.2.8.148"; destination host is: "test.152.com":8021; >>> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) >>> > at org.apache.hadoop.ipc.Client.call(Client.java:1410) >>> > at org.apache.hadoop.ipc.Client.call(Client.java:1359) >>> > at >>> > >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) >>> > at com.sun.proxy.$Proxy23.startContainers(Unknown Source) >>> > at >>> > >>> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96) >>> > at >>> > >>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118) >>> > at >>> > >>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249) >>> > at >>> > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> > at >>> > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> > at java.lang.Thread.run(Thread.java:744) >>> >>> >>> -- >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >>> to >>> which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified >>> that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender >>> immediately >>> and delete it from your system. Thank You. >>> >> >> >
