Thank for the response. I have removed rpc.protection parameter in all of my configuration and now I am getting an error as below:-
Any Hint on whats going on here 13/11/14 10:10:47 INFO mapreduce.Job: Task Id : attempt_1384339616944_0002_m_000026_0, Status : FAILED Container launch failed for container_1384339616944_0002_01_000029 : java.net.SocketTimeoutException: Call From U1204D32/40.221.95.97 to U1204D32.blr.in.as.airbus.corp:45003 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/ 40.221.95.97:45003]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:749) at org.apache.hadoop.ipc.Client.call(Client.java:1351) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy30.startContainers(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:151) at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/ 40.221.95.97:45003] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.FilterInputStream.read(FilterInputStream.java:133) at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891) 13/11/14 10:10:50 INFO mapreduce.Job: map 72% reduce 24% 13/11/14 10:19:13 INFO mapreduce.Job: map 76% reduce 24% 13/11/14 10:19:55 INFO mapreduce.Job: map 78% reduce 24% 13/11/14 10:20:10 INFO mapreduce.Job: map 80% reduce 24% 13/11/14 10:20:29 INFO mapreduce.Job: map 81% reduce 24% 13/11/14 10:21:50 INFO mapreduce.Job: map 82% reduce 24% 13/11/14 10:22:01 INFO mapreduce.Job: map 83% reduce 24% 13/11/14 10:22:10 INFO mapreduce.Job: map 84% reduce 24% 13/11/14 10:22:13 INFO mapreduce.Job: map 85% reduce 24% 13/11/14 10:22:16 INFO mapreduce.Job: map 86% reduce 24% 13/11/14 10:22:27 INFO mapreduce.Job: map 86% reduce 28% 13/11/14 10:22:30 INFO mapreduce.Job: map 86% reduce 29% 13/11/14 10:28:30 INFO mapreduce.Job: map 88% reduce 29% 13/11/14 10:28:35 INFO mapreduce.Job: map 90% reduce 29% 13/11/14 10:28:37 INFO mapreduce.Job: map 97% reduce 29% 13/11/14 10:28:43 INFO mapreduce.Job: map 98% reduce 29% 13/11/14 10:28:45 INFO mapreduce.Job: map 99% reduce 29% 13/11/14 10:28:48 INFO mapreduce.Job: map 100% reduce 29% 13/11/14 10:28:55 INFO mapreduce.Job: map 100% reduce 33% 13/11/14 10:29:06 INFO mapreduce.Job: map 100% reduce 67% 13/11/14 10:29:27 INFO mapreduce.Job: map 100% reduce 100% 13/11/14 10:29:33 INFO mapreduce.Job: Job job_1384339616944_0002 completed successfully 13/11/14 10:29:38 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server 13/11/14 10:29:55 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server 13/11/14 10:29:56 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server 13/11/14 10:29:56 ERROR security.UserGroupInformation: PriviledgedActionException as:prrekapaee9f (auth:SIMPLE) cause:java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220) at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159) at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) [Error] Exception: java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220) at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159) at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331) at org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:368) at org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511) at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756) at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753) at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361) at org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:407) at org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:855) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:835) at com.eads.dcgo.uc4.DCGoUC4.run(DCGoUC4.java:593) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at com.eads.dcgo.uc4.DCGoUC4.main(DCGoUC4.java:694) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) On 13 Nov 2013 20:11, "Daryn Sharp" <[email protected]> wrote: > "No common protection layer between server and client " likely means the > host for job submission does not have hadoop.rpc.protection=privacy. In > order for QOP to work, all client hosts (DN & others used to access the > cluster) must have an identical setting. > > A few quick questions: I'm assuming you mis-posted your configs and the > protection setting isn't really commented out? Your configs don't show > security being enabled, but you do have it enabled, correct? Otherwise QOP > shouldn't apply. Perhaps a bit obvious, but did you restart your NN after > changing the QOP? Since your defaultFS is just "master", are you using HA? > > It's a bit concerning that you aren't consistently receiving the > mismatch error. Is the client looping on retries and then you get timeouts > after 5 attempts? If yes, we've got a major bug. 5 is the default number > of RPC readers which handle SASL auth which means the protection mismatch > is killing off the reader threads and rendering the NN unusable. This > shouldn't be possible, but what does your NN log show? > > Daryn > > On Nov 13, 2013, at 6:05 AM, rab ra <[email protected]> wrote: > > > > Hello, > > I am facing a problem in using Hadoop RPC encryption while transfer > feature in hadoop 2.2.0. I have 3 node cluster > > > Service running in node 1 (master) > Resource manager > Namenode > DataNode > SecondaryNamenode > > Service running in slaves ( node 2 & 3) > NodeManager > > > > I am trying to make data transfer between master and slave secure. For > that, I wanted to use data encryption over wire (RPC encryption) feature of > hadoop 2.2.0 > > When I ran the code, I get the below exception > > Caused by: java.net.SocketTimeoutException: 60000 millis timeout while > waiting for channel to be ready for read. > > > In another run, I saw in log the following error > > No common protection layer between server and client > > Not sure whether my configuration is inline with what I want to achieve. > > Can someone give me some hint on where I am going wrong? > > By the way, I have the below configuration setting in all of these nodes > > Core-site.xml > > <configuration> > > <property> > <name>fs.defaultFS</name> > <value>hdfs://master:8020</value> > </property> > > <property> > <name>hadoop.tmp.dir</name> > <value>/tmp</value> > </property> > <!-- > <property> > <name>hadoop.rpc.protection</name> > <value>privacy</value> > </property> > --> > <property> > <name>io.file.buffer.size</name> > <value>131072</value> > </property> > > </configuration> > > Hdfs-site.xml > <configuration> > > <property> > <name>dfs.replication</name> > <value>1</value> > </property> > > <property> > <name>dfs.name.dir</name> > <value>/app/hadoop/dfs-2.2.0/name</value> > </property> > > <property> > <name>dfs.data.dir</name> > <value>/app/hadoop/dfs-2.2.0/data</value> > </property> > > <property> > <name>dfs.encrypt.data.transfer</name> > <value>true</value> > </property> > > <property> > <name>dfs.encrypt.data.transfer.algorithm</name> > <value>rc4</value> > </property> > > <property> > <name>dfs.block.access.token.enable</name> > <value>true</value> > </property> > > </configuration> > > Mapred-site.xml > > <configuration> > > <property> > <name>mapreduce.framework.name</name> > <value>yarn</value> > </property> > <!-- > <property> > <name>mapreduce.jobtracker.address</name> > <value>master:8032</value> > </property> > --> > <property> > <name>mapreduce.tasktracker.map.tasks.maximum</name> > <value>1</value> > </property> > > <property> > <name>mapreduce.tasktracker.reduce.tasks.maximum</name> > <value>1</value> > </property> > > <property> > <name>mapreduce.map.speculative</name> > <value>false</value> > </property> > > <property> > <name>mapreduce.reduce.speculative</name> > <value>false</value> > </property> > > <property> > <name>mapreduce.map.java.opts</name> > <value>-Xmx1024m</value> > </property> > > </configuration> > > > Yarn-site.xml > > <configuration> > > <property> > <name>yarn.resourcemanager.hostname</name> > <value>master</value> > </property> > > <property> > <name>yarn.log-aggregation-enable</name> > <value>true</value> > </property> > > <property> > <name>yarn.nodemanager.aux-services</name> > <value>mapreduce_shuffle</value> > </property> > > <property> > <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> > <value>org.apache.hadoop.mapred.ShuffleHandler</value> > </property> > > </configuration> > > > > With thanks and regards > Rab > > > > > >
