[
https://issues.apache.org/jira/browse/HDFS-11367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15844089#comment-15844089
]
Manjunath Anand commented on HDFS-11367:
----------------------------------------
HDFS supports single writer per file at any given point of time as can be seen
in the JIRA https://issues.apache.org/jira/browse/HDFS-8177.
Even if you use a single thread per file and if that thread creates a file and
using the returned OutputStream writes data to file then the file will be
having an under construction feature and a lease at the time of creating the
file which will be cached. If the same thread tries to do an append without
closing the OutputStream then the HDFS code finds that the file is
underconstruction with a valid lease and doesnot another lease to be created by
the same thread.
Once you get the OutputStream object from your getOutputStream method and once
done writing data, close the OutputStream . This will make sure the
underconstruction feature of the file and lease is removed. Then calling the
getOutputStream later will result in append and would not result in this kind
of error.
> AlreadyBeingCreatedException "current leaseholder is trying to recreate file"
> when trying to append to file
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDFS-11367
> URL: https://issues.apache.org/jira/browse/HDFS-11367
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.5.0
> Environment: Red Hat Enterprise Linux Server release 6.8
> Reporter: Dmitry Goldenberg
> Assignee: Manjunath Anand
>
> We have code which creates a file in HDFS and continuously appends lines to
> the file, then closes the file at the end. This is done by a single dedicated
> thread.
> We specifically instrumented the code to make sure only one 'client'/thread
> ever writes to the file because we were seeing "current leaseholder is trying
> to recreate file" errors.
> For some background see this for example:
> https://community.cloudera.com/t5/Storage-Random-Access-HDFS/How-to-append-files-to-HDFS-with-Java-quot-current-leaseholder/m-p/41369
> This issue is very critical to us as any error terminates a mission critical
> application in production.
> Intermittently, we see the below exception, regardless of what our code is
> doing which is create the file, keep appending, then close:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException):
> failed to create file /data/records_20170125_1.txt for
> DFSClient_NONMAPREDUCE_-167421175_1 for client 1XX.2XX.1XX.XXX because
> current leaseholder is trying to recreate file.
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:3075)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2905)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:3189)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:3153)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:612)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.append(AuthorizationProviderProxyClientProtocol.java:125)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:414)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1767)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1411)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy24.append(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy24.append(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.append(ClientNamenodeProtocolTranslatorPB.java:282)
> at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1586)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1626)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1614)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:313)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:309)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:309)
> at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1161)
> at com.myco.MyAppender.getOutputStream(MyAppender.java:147)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
