Hi everyone,
I set Space Quotas for the amount of space on HDFS.
But I have some questions.
■Question
Q1：Why does diskspace become MByet when the file of KByte is input?(1024
calculations in HDFS)
Q2：Is there a person who has information for this problem that quota exceed
exception,but it creates file of size 0?
Q3：Why did input file's capacity exceed the limit(space quota)?
■Details of question
★First time
Even when I put file which was smaller than the capacity set in Space Quota,
it became an error.
The input file's capacity which I put is 64000=64kbyte.
・input file
$ ls -ltr
$ -rw-r--r-- 1 root root 64000 Jan 21 04:42 xaa
$ du -h xaa
68K xaa
・hdfs-site.xml(replication)
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
・Space Quota
$ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
$ ./bin/hadoop fs -count -q /user/root/input/
none inf 192000 192000 1
0 0 hdfs://drbd-test-vm03/user/root/input
$ ./bin/hadoop dfs -put input/xaa /user/root/input/
10/01/21 19:35:58 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.hdfs.protocol.QuotaExceededException:
org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
/user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
quota=192000 diskspace=134217728
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:58)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2755)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2046)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2232)
Caused by: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
/user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
quota=192000 diskspace=134217728
at
org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:161)
at
org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateNumItemsInTree(INodeDirectoryWithQuota.java:134)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:859)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:265)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1427)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1274)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2873)
... 3 more
10/01/21 19:35:58 WARN hdfs.DFSClient: Error Recovery for block null bad
datanode[0] nodes == null
10/01/21 19:35:58 WARN hdfs.DFSClient: Could not get block locations. Source
file "/user/root/input/xaa" - Aborting...
put: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
/user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
quota=192000 diskspace=134217728
When I saw the above error messages,it was described as diskspace=134217728.
Perhaps, I think that it is "134217728/1024^2=128MByte".
I understood that this is why The quota(192000=192Kbyte) of /user/root/input
was exceeded.
Q1：Why does diskspace become MByet when the file of KByte is input?(1024
calculations in HDFS)
[r...@drbd-test-vm03 current]# ./bin/hadoop dfs -lsr /user/root/input/xaa
-rw-r--r-- 2 root supergroup 0 2010-01-21 19:35 /user/root/input/xaa
Quota exceed exception,but it creates file of size 0.
It seems that this is the similar problem as the problem described in following
URL.
http://issues.apache.org/jira/browse/HDFS-172
Has not this problem solved yet?
Q2：Is there a person who has information for this problem that quota exceed
exception,but it creates file of size 0?
★Since the second times
The result was different the first time and the second times.
It doesn't make an error even if it exceeds capacity since the second times.
Imput file's capacity is 64000=64k.
Reprication figure is 2.
Calculated result
102400-(64000×2)=-25600<-- Why does it become a "-" mark without becoming an
error?
I thought that the space quota is a limit on the number of bytes used by files
at that directory.
However, the result shows the thing to which my assumption is wrong.
Q3：Why did input file's capacity exceed the limit(space quota)?
[r...@drbd-test-vm03 current]# ./bin/hadoop dfsadmin -setSpaceQuota 100K
/user/root/input/
[r...@drbd-test-vm03 current]# ./bin/hadoop fs -count -q /user/root/input/
none inf 102400 102400 1
0 0 hdfs://drbd-test-vm03/user/root/input
[r...@drbd-test-vm03 current]# ./bin/hadoop dfs -put xaa /user/root/input/
[r...@drbd-test-vm03 current]# ./bin/hadoop fs -count -q /user/root/input/
none inf 102400 -25600 1
1 64000 hdfs://drbd-test-vm03/user/root/input
Best regards,
Tadashi.
