[
https://issues.apache.org/jira/browse/HDFS-13164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372114#comment-16372114
]
Xiao Chen edited comment on HDFS-13164 at 2/21/18 10:22 PM:
------------------------------------------------------------
The 0-sized file should be fine, as long as we make sure the client closes it
correctly, and no resource leakages in the face of an exception.
The separation of block and file quota isn't HDFS specific - I tested this on
Linux (CentOs 7.3) and got a similar result (without the leakages of course).
{noformat}
[systest@xiaog-4 share]$ quota
Disk quotas for user systest (uid 2000):
Filesystem blocks quota limit grace files quota limit grace
/dev/loop0 5* 2 5 7days 6 0 0
[systest@xiaog-4 share]$ cp /etc/hosts ./c6
loop0: write failed, user block limit reached.
cp: error writing ‘./c6’: Disk quota exceeded
cp: failed to extend ‘./c6’: Disk quota exceeded
[systest@xiaog-4 share]$ ls -rtl
total 5
-rw-rw-r-- 1 systest quotagrp 0 Feb 21 14:09 f1
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c1
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c2
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c3
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c4
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c5
-rw-r--r-- 1 systest quotagrp 0 Feb 21 14:11 c6
[systest@xiaog-4 share]$ cat >> c6
whatever
cat: write error: Disk quota exceeded
[systest@xiaog-4 share]$ ls -rtl
total 5
-rw-rw-r-- 1 systest quotagrp 0 Feb 21 14:09 f1
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c1
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c2
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c3
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c4
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c5
-rw-r--r-- 1 systest quotagrp 0 Feb 21 14:11 c6
{noformat}
Locally have some unit tests ready, currently working on a proper fix, will
post soon.
Notably this issue probably also applies to {{DFSStripedOutputStream}}. Given
the implementation and fix would be separate. Created HDFS-13177 for that.
was (Author: xiaochen):
The 0-sized file should be fine, as long as we make sure the client closes it
correctly, and no resource leakages in the face of an exception.
The separation of block and file quota isn't HDFS specific - I tested this on
Linux (CentOs 7.3) and got a similar result (without the leakages of course).
{noformat}
[systest@xiaog-4 share]$ quota
Disk quotas for user systest (uid 2000):
Filesystem blocks quota limit grace files quota limit grace
/dev/loop0 5* 2 5 7days 6 0 0
[systest@xiaog-4 share]$ cp /etc/hosts ./c6
loop0: write failed, user block limit reached.
cp: error writing ‘./c6’: Disk quota exceeded
cp: failed to extend ‘./c6’: Disk quota exceeded
[systest@xiaog-4 share]$ ls -rtl
total 5
-rw-rw-r-- 1 systest quotagrp 0 Feb 21 14:09 f1
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c1
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c2
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c3
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c4
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c5
-rw-r--r-- 1 systest quotagrp 0 Feb 21 14:11 c6
[systest@xiaog-4 share]$ cat >> c6
whatever
cat: write error: Disk quota exceeded
[systest@xiaog-4 share]$ ls -rtl
total 5
-rw-rw-r-- 1 systest quotagrp 0 Feb 21 14:09 f1
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c1
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c2
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c3
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c4
-rw-r--r-- 1 systest quotagrp 285 Feb 21 14:10 c5
-rw-r--r-- 1 systest quotagrp 0 Feb 21 14:11 c6
{noformat}
Locally have some unit tests ready, currently working on a proper fix, will
post soon.
Notably this issue probably also applies to {{DFSStripedOutputStream}}. Given
the implementation and fix would be separate, I'll create a separate jira for
that.
> File not closed if append fail with DSQuotaExceededException
> ------------------------------------------------------------
>
> Key: HDFS-13164
> URL: https://issues.apache.org/jira/browse/HDFS-13164
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.6.5
> Reporter: Xiao Chen
> Assignee: Xiao Chen
> Priority: Major
>
> This is found during yarn log aggregation but theoretically could happen to
> any client.
> If the dir's space quota is exceeded, the following would happen when a file
> is created:
> - client {{startFile}} rpc to NN, gets a {{DFSOutputStream}}.
> - writing to the stream would trigger the streamer to {{getAdditionalBlock}}
> rpc to NN, which would get the DSQuotaExceededException
> - client closes the stream
>
> The fact that this would leave a 0-sized (or whatever size left in the
> quota) file in HDFS is beyond the scope of this jira. However, the file would
> be left in openforwrite status (shown in {{fsck -openforwrite)}} at least,
> and could potentially leak leaseRenewer too.
> This is because in the close implementation,
> # {{isClosed}} is first checked, and the close call will be a no-op if
> {{isClosed == true}}.
> # {{flushInternal}} checks {{isClosed}}, and throws the exception right away
> if true
> {{isClosed}} does this: {{return closed || getStreamer().streamerClosed;}}
> When the disk quota is reached, {{getAdditionalBlock}} will throw when the
> streamer calls. Because the streamer runs in a separate thread, at the time
> the client calls close on the stream, the streamer may or may not have
> reached the Quota exception. If it has, then due to #1, the close call on the
> stream will be no-op. If it hasn't, then due to #2 the {{completeFile}} logic
> will be skipped.
> {code:java}
> protected synchronized void closeImpl() throws IOException {
> if (isClosed()) {
> IOException e = lastException.getAndSet(null);
> if (e == null)
> return;
> else
> throw e;
> }
> try {
> flushBuffer(); // flush from all upper layers
> ...
> flushInternal(); // flush all data to Datanodes
> // get last block before destroying the streamer
> ExtendedBlock lastBlock = getStreamer().getBlock();
> try (TraceScope ignored =
> dfsClient.getTracer().newScope("completeFile")) {
> completeFile(lastBlock);
> }
> } catch (ClosedChannelException ignored) {
> } finally {
> closeThreads(true);
> }
> }
> {code}
> Log snippets:
> {noformat}
> 2018-02-16 15:59:32,916 DEBUG org.apache.hadoop.hdfs.DFSClient: DataStreamer
> Quota Exception
> org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota
> of /DIR is exceeded: quota = 2000000 B = 1.91 MB but diskspace consumed =
> 404139552 B = 385.42 MB
> at
> org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyDiskspaceQuota(DirectoryWithQuotaFeature.java:149)
> at
> org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:159)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:2124)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1991)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1966)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:463)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3896)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3484)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:686)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:217)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:506)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2222)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2220)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1833)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1626)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:788)
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.DSQuotaExceededException):
> The DiskSpace quota of /tmp/logs/systest/logs is exceeded: quota = 2000000 B
> = 1.91 MB but diskspace consumed = 404139552 B = 385.42 MB
> at
> org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyDiskspaceQuota(DirectoryWithQuotaFeature.java:149)
> at
> org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:159)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:2124)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1991)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1966)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:463)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3896)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3484)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:686)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:217)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:506)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2222)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2220)
> at org.apache.hadoop.ipc.Client.call(Client.java:1504)
> at org.apache.hadoop.ipc.Client.call(Client.java:1441)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy82.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:423)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy83.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1830)
> ... 2 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
