Any hint ? On 14 Nov 2013 11:45, "rab ra" <[email protected]> wrote:
> Thank for the response. > > I have removed rpc.protection parameter in all of my configuration and now > I am getting an error as below:- > > > > Any Hint on whats going on here > > > > 13/11/14 10:10:47 INFO mapreduce.Job: Task Id : > attempt_1384339616944_0002_m_000026_0, Status : FAILED > > Container launch failed for container_1384339616944_0002_01_000029 : > java.net.SocketTimeoutException: Call From U1204D32/40.221.95.97 to > U1204D32.blr.in.as.airbus.corp:45003 failed on socket timeout exception: > java.net.SocketTimeoutException: 60000 millis timeout while waiting for > channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected > local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/ > 40.221.95.97:45003]; For more details see: > http://wiki.apache.org/hadoop/SocketTimeout > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) > > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:749) > > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > > at com.sun.proxy.$Proxy30.startContainers(Unknown Source) > > at > org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96) > > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:151) > > at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) > > Caused by: java.net.SocketTimeoutException: 60000 millis timeout while > waiting for channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected > local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/ > 40.221.95.97:45003] > > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) > > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) > > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) > > at java.io.FilterInputStream.read(FilterInputStream.java:133) > > at java.io.FilterInputStream.read(FilterInputStream.java:133) > > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457) > > at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > > at java.io.BufferedInputStream.read(BufferedInputStream.java:254) > > at java.io.DataInputStream.readInt(DataInputStream.java:387) > > at > org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995) > > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891) > > > > 13/11/14 10:10:50 INFO mapreduce.Job: map 72% reduce 24% > > 13/11/14 10:19:13 INFO mapreduce.Job: map 76% reduce 24% > > 13/11/14 10:19:55 INFO mapreduce.Job: map 78% reduce 24% > > 13/11/14 10:20:10 INFO mapreduce.Job: map 80% reduce 24% > > 13/11/14 10:20:29 INFO mapreduce.Job: map 81% reduce 24% > > 13/11/14 10:21:50 INFO mapreduce.Job: map 82% reduce 24% > > 13/11/14 10:22:01 INFO mapreduce.Job: map 83% reduce 24% > > 13/11/14 10:22:10 INFO mapreduce.Job: map 84% reduce 24% > > 13/11/14 10:22:13 INFO mapreduce.Job: map 85% reduce 24% > > 13/11/14 10:22:16 INFO mapreduce.Job: map 86% reduce 24% > > 13/11/14 10:22:27 INFO mapreduce.Job: map 86% reduce 28% > > 13/11/14 10:22:30 INFO mapreduce.Job: map 86% reduce 29% > > 13/11/14 10:28:30 INFO mapreduce.Job: map 88% reduce 29% > > 13/11/14 10:28:35 INFO mapreduce.Job: map 90% reduce 29% > > 13/11/14 10:28:37 INFO mapreduce.Job: map 97% reduce 29% > > 13/11/14 10:28:43 INFO mapreduce.Job: map 98% reduce 29% > > 13/11/14 10:28:45 INFO mapreduce.Job: map 99% reduce 29% > > 13/11/14 10:28:48 INFO mapreduce.Job: map 100% reduce 29% > > 13/11/14 10:28:55 INFO mapreduce.Job: map 100% reduce 33% > > 13/11/14 10:29:06 INFO mapreduce.Job: map 100% reduce 67% > > 13/11/14 10:29:27 INFO mapreduce.Job: map 100% reduce 100% > > 13/11/14 10:29:33 INFO mapreduce.Job: Job job_1384339616944_0002 completed > successfully > > 13/11/14 10:29:38 INFO mapred.ClientServiceDelegate: Application state is > completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history > server > > 13/11/14 10:29:55 INFO mapred.ClientServiceDelegate: Application state is > completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history > server > > 13/11/14 10:29:56 INFO mapred.ClientServiceDelegate: Application state is > completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history > server > > 13/11/14 10:29:56 ERROR security.UserGroupInformation: > PriviledgedActionException as:prrekapaee9f (auth:SIMPLE) > cause:java.io.IOException: > org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): > java.lang.NullPointerException > > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220) > > at > org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159) > > at > org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281) > > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) > > > > [Error] Exception: > > java.io.IOException: > org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): > java.lang.NullPointerException > > at > org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220) > > at > org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159) > > at > org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281) > > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) > > > > at > org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331) > > at > org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:368) > > at > org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511) > > at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756) > > at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > > at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753) > > at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361) > > at > org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:407) > > at > org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:855) > > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:835) > > at com.eads.dcgo.uc4.DCGoUC4.run(DCGoUC4.java:593) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > > at com.eads.dcgo.uc4.DCGoUC4.main(DCGoUC4.java:694) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > On 13 Nov 2013 20:11, "Daryn Sharp" <[email protected]> wrote: > >> "No common protection layer between server and client " likely means the >> host for job submission does not have hadoop.rpc.protection=privacy. In >> order for QOP to work, all client hosts (DN & others used to access the >> cluster) must have an identical setting. >> >> A few quick questions: I'm assuming you mis-posted your configs and the >> protection setting isn't really commented out? Your configs don't show >> security being enabled, but you do have it enabled, correct? Otherwise QOP >> shouldn't apply. Perhaps a bit obvious, but did you restart your NN after >> changing the QOP? Since your defaultFS is just "master", are you using HA? >> >> It's a bit concerning that you aren't consistently receiving the >> mismatch error. Is the client looping on retries and then you get timeouts >> after 5 attempts? If yes, we've got a major bug. 5 is the default number >> of RPC readers which handle SASL auth which means the protection mismatch >> is killing off the reader threads and rendering the NN unusable. This >> shouldn't be possible, but what does your NN log show? >> >> Daryn >> >> On Nov 13, 2013, at 6:05 AM, rab ra <[email protected]> wrote: >> >> >> >> Hello, >> >> I am facing a problem in using Hadoop RPC encryption while transfer >> feature in hadoop 2.2.0. I have 3 node cluster >> >> >> Service running in node 1 (master) >> Resource manager >> Namenode >> DataNode >> SecondaryNamenode >> >> Service running in slaves ( node 2 & 3) >> NodeManager >> >> >> >> I am trying to make data transfer between master and slave secure. For >> that, I wanted to use data encryption over wire (RPC encryption) feature of >> hadoop 2.2.0 >> >> When I ran the code, I get the below exception >> >> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while >> waiting for channel to be ready for read. >> >> >> In another run, I saw in log the following error >> >> No common protection layer between server and client >> >> Not sure whether my configuration is inline with what I want to achieve. >> >> Can someone give me some hint on where I am going wrong? >> >> By the way, I have the below configuration setting in all of these nodes >> >> Core-site.xml >> >> <configuration> >> >> <property> >> <name>fs.defaultFS</name> >> <value>hdfs://master:8020</value> >> </property> >> >> <property> >> <name>hadoop.tmp.dir</name> >> <value>/tmp</value> >> </property> >> <!-- >> <property> >> <name>hadoop.rpc.protection</name> >> <value>privacy</value> >> </property> >> --> >> <property> >> <name>io.file.buffer.size</name> >> <value>131072</value> >> </property> >> >> </configuration> >> >> Hdfs-site.xml >> <configuration> >> >> <property> >> <name>dfs.replication</name> >> <value>1</value> >> </property> >> >> <property> >> <name>dfs.name.dir</name> >> <value>/app/hadoop/dfs-2.2.0/name</value> >> </property> >> >> <property> >> <name>dfs.data.dir</name> >> <value>/app/hadoop/dfs-2.2.0/data</value> >> </property> >> >> <property> >> <name>dfs.encrypt.data.transfer</name> >> <value>true</value> >> </property> >> >> <property> >> <name>dfs.encrypt.data.transfer.algorithm</name> >> <value>rc4</value> >> </property> >> >> <property> >> <name>dfs.block.access.token.enable</name> >> <value>true</value> >> </property> >> >> </configuration> >> >> Mapred-site.xml >> >> <configuration> >> >> <property> >> <name>mapreduce.framework.name</name> >> <value>yarn</value> >> </property> >> <!-- >> <property> >> <name>mapreduce.jobtracker.address</name> >> <value>master:8032</value> >> </property> >> --> >> <property> >> <name>mapreduce.tasktracker.map.tasks.maximum</name> >> <value>1</value> >> </property> >> >> <property> >> <name>mapreduce.tasktracker.reduce.tasks.maximum</name> >> <value>1</value> >> </property> >> >> <property> >> <name>mapreduce.map.speculative</name> >> <value>false</value> >> </property> >> >> <property> >> <name>mapreduce.reduce.speculative</name> >> <value>false</value> >> </property> >> >> <property> >> <name>mapreduce.map.java.opts</name> >> <value>-Xmx1024m</value> >> </property> >> >> </configuration> >> >> >> Yarn-site.xml >> >> <configuration> >> >> <property> >> <name>yarn.resourcemanager.hostname</name> >> <value>master</value> >> </property> >> >> <property> >> <name>yarn.log-aggregation-enable</name> >> <value>true</value> >> </property> >> >> <property> >> <name>yarn.nodemanager.aux-services</name> >> <value>mapreduce_shuffle</value> >> </property> >> >> <property> >> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> >> <value>org.apache.hadoop.mapred.ShuffleHandler</value> >> </property> >> >> </configuration> >> >> >> >> With thanks and regards >> Rab >> >> >> >> >> >>
