Hi, Please also check for coalesced RDD. I encountered the same error while writing a coalesced rdd/df to HDFS. If this is the case, please use repartition instead.
Sent from OnePlus 3 Thanks & Regards, Gaurav Kumar On Oct 17, 2016 11:22 AM, "Zhang Jianfeng" <jzhang...@gmail.com> wrote: Thanks Rakesh for your kind help. Actually during the job only one reducer result file (for example part-r-2) had this error, other reducers worked well. Best Regards, Jian Feng 2016-10-17 11:49 GMT+08:00 Rakesh Radhakrishnan <rake...@apache.org>: > Hi Jian Feng, > > Could you please check your code and see any possibilities of simultaneous > access to the same file. Mostly this situation happens when multiple > clients tries to access the same file. > > Code Reference:- https://github.com/apache/hadoop/blob/branch-2. > 2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/ > hadoop/hdfs/server/namenode/FSNamesystem.java#L2737 > > Best Regards, > Rakesh > Intel > > On Mon, Oct 17, 2016 at 7:16 AM, Zhang Jianfeng <jzhang...@gmail.com> > wrote: > >> Hi , >> >> I hit an wired error. On our hadoop cluster (2.2.0), occasionally a >> LeaseExpiredException is thrown. >> >> The stacktrace is as below: >> >> >> *org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): >> No lease on /user/biadmin/analytic‐root/SX5XPWPPDPQH/.executions/.at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2737)* >> >> *at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:2801)* >> >> *at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2783)* >> >> *at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.com >> <http://namenode.NameNodeRpcServer.com>plete(NameNodeRpcServer.java:611)* >> >> *at >> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:428)* >> >> *at >> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59586)* >> >> *at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)* >> >> *at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)* >> >> *at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)* >> >> *at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)* >> >> *at >> java.security.AccessController.doPrivileged(AccessController.java:310)* >> >> *at javax.security.auth.Subject.do >> <http://javax.security.auth.Subject.do>As(Subject.java:573)* >> >> *at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)* >> >> *at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)* >> >> *at org.apache.hadoop.ipc.Client.call(Client.java:1347)* >> >> *at org.apache.hadoop.ipc.Client.call(Client.java:1300)* >> >> *at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)* >> >> *at $Proxy7.complete(Unknown Source)* >> >> *at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)* >> >> *at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)* >> >> *at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)* >> >> at java.lang.reflect.Method.invoke(Method.java:611) >> >> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth >> od(RetryInvocationHandler.java:186) >> >> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret >> ryInvocationHandler.java:102) >> >> at $Proxy7.complete(Unknown Source) >> >> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran >> slatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:371) >> >> at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutpu >> tStream.java:1894) >> >> at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream >> .java:1881) >> >> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close( >> FSDataOutputStream.java:71) >> >> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputSt >> ream.java:104) >> >> at java.io.FilterOutputStream.close(FilterOutputStream.java:154) >> >> Any help will be appreciated! >> >> -- >> Best Regards, >> Jian Feng >> > > -- Best Regards, Jian Feng
