Hi everyone, I have the problems about HDFS Quota.
I set HDFS Quota. I thought that it became an error when the quota size was exceeded. However, it became "-" mark without becoming an error. Why didn't the Space Quota setting become effective? Is there the person who has information about this problems? Best regards, Tadashi > -----Original Message----- > From: tate...@nttdata.co.jp [mailto:tate...@nttdata.co.jp] > Sent: Tuesday, January 26, 2010 4:05 PM > To: hdfs-user@hadoop.apache.org > Subject: RE: HDFS Quota > > Hi Boris, > > Thank you for your reply. > > > From your example it looks like you have namespace quota (not diskspace), > > i.e. quota for quantities of files (not size) > I used following command. > > $ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/ > > I thought this was a limit on total size(192000) of all the files > under the directory tree(/user/root/input/). > > Does quota for quantities of files (not size) mean Name Quotas? > > I found the following sentences. > Name Quotas is a hard limit on the number of file and directory. > Space Quotas is a hard limit on the number of bytes used by files. > > Sorry,I'm confused about the difference between namespace and diskspace. > > > Yes, I think this is by design. When HDFS creates a file, it doesn't know > > how big the file will be. So diskspace quota is checked on writing to > the > > file (not on creation). > I understood that. > However, I think that the system recognize that the file exists even by > 0 bytes. > Therefore, should I take the action deleted on the application side when > it becomes 0 bytes? > > > What is your block size. I don't think it make sense to put quota for > less > > then a blocksize. > My block size is 64MByte(default setting). > I understood that I should have set the quota larger than the block size. > > File size is 134217728/1024^2=128MByte.(larger than the block > size(64MByte)) > Quota size is 256MByte(256*1024^2=268435456).(larger than the block > size(64MByte)) > Number of replication is 2. > > # ls -ltr xaa > -rw-r--r-- 1 root root 134217728 Jan 26 01:45 xaa > # ls -ltr xab > -rw-r--r-- 1 root root 134217728 Jan 26 01:45 xab > # ./bin/hadoop dfsadmin -setSpaceQuota 256m /user/root/input/ > # ./bin/hadoop fs -count -q /user/root/input > none inf 268435456 268435456 > 1 0 0 hdfs://drbd-test-vm03/user/root/input > # ./bin/hadoop dfs -put xaa /user/root/input/ > # ./bin/hadoop fs -count -q /user/root/input > none inf 268435456 0 > 1 1 134217728 hdfs://drbd-test-vm03/user/root/input > # ./bin/hadoop dfs -put xab /user/root/input/ > # ./bin/hadoop fs -count -q /user/root/input > none inf 268435456 -268435456 > 1 2 268435456 hdfs://drbd-test-vm03/user/root/input > > I thought that it became an error when the quota size was exceeded. > However, it was a "-" mark without becoming an error. > Why didn't the Space Quota setting(hard limit on the number of bytes) become > effective? > > Best regards, > Tadashi. > > -----Original Message----- > > From: Boris Shkolnik [mailto:bo...@yahoo-inc.com] > > Sent: Tuesday, January 26, 2010 5:20 AM > > To: hdfs-user@hadoop.apache.org > > Subject: Re: HDFS Quota > > > > >>> /user/root/input is exceeded: namespace quota=-1 file count=2, > > From your example it looks like you have namespace quota (not diskspace), > > i.e. quota for quantities of files (not size) > > > > > > > Quota exceed exception,but it creates file of size 0. > > Yes, I think this is by design. When HDFS creates a file, it doesn't know > > how big the file will be. So diskspace quota is checked on writing to > the > > file (not on creation). > > > > >Q3：Why did input file's capacity exceed the limit(space quota)? > > What is your block size. I don't think it make sense to put quota for > less > > then a blocksize. > > > > Boris. > > > > On 1/25/10 1:01 AM, "tate...@nttdata.co.jp" <tate...@nttdata.co.jp> > wrote: > > > > > Hi everyone, > > > > > > I set Space Quotas for the amount of space on HDFS. > > > But I have some questions. > > > > > > ■Question > > > Q1：Why does diskspace become MByet when the file of KByte is input?(1024 > > > calculations in HDFS) > > > Q2：Is there a person who has information for this problem that quota > exceed > > > exception,but it creates file of size 0? > > > Q3：Why did input file's capacity exceed the limit(space quota)? > > > > > > > > > ■Details of question > > > ★First time > > > Even when I put file which was smaller than the capacity set in Space > Quota, > > > it became an error. > > > > > > The input file's capacity which I put is 64000=64kbyte. > > > > > > ・input file > > > $ ls -ltr > > > $ -rw-r--r-- 1 root root 64000 Jan 21 04:42 xaa > > > $ du -h xaa > > > 68K xaa > > > > > > ・hdfs-site.xml(replication) > > > <property> > > > <name>dfs.replication</name> > > > <value>2</value> > > > </property> > > > > > > ・Space Quota > > > $ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/ > > > $ ./bin/hadoop fs -count -q /user/root/input/ > > > none inf 192000 192000 > > 1 > > > 0 0 hdfs://drbd-test-vm03/user/root/input > > > $ ./bin/hadoop dfs -put input/xaa /user/root/input/ > > > 10/01/21 19:35:58 WARN hdfs.DFSClient: DataStreamer Exception: > > > org.apache.hadoop.hdfs.protocol.QuotaExceededException: > > > org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of > > > /user/root/input is exceeded: namespace quota=-1 file count=2, > diskspace > > > quota=192000 diskspace=134217728 > > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > > Method) > > > at > > > > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo > rAc > > cesso > > > rImpl.java:39) > > > at > > > > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo > nst > > ructo > > > rAccessorImpl.java:27) > > > at > > java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > > at > > > > > > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteExcep > tio > > n.jav > > > a:96) > > > at > > > > > > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteExce > pti > > on.ja > > > va:58) > > > at > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock( > DFS > > Clien > > > t.java:2875) > > > at > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream > (DF > > SClie > > > nt.java:2755) > > > at > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient > .ja > > va:20 > > > 46) > > > at > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSC > lie > > nt.ja > > > va:2232) > > > Caused by: org.apache.hadoop.ipc.RemoteException: > > > org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of > > > /user/root/input is exceeded: namespace quota=-1 file count=2, > diskspace > > > quota=192000 diskspace=134217728 > > > at > > > > > > org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQ > uot > > a(INo > > > deDirectoryWithQuota.java:161) > > > at > > > > > > org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateN > umI > > temsI > > > nTree(INodeDirectoryWithQuota.java:134) > > > at > > > > > > org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirec > tor > > y.jav > > > a:859) > > > at > > > > > > org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirector > y.j > > ava:2 > > > 65) > > > at > > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNa > mes > > ystem > > > .java:1427) > > > at > > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock > (FS > > Names > > > ystem.java:1274) > > > at > > > > > > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java > :42 > > 2) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > > at > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > ava > > :39) > > > at > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > orI > > mpl.j > > > ava:25) > > > at java.lang.reflect.Method.invoke(Method.java:597) > > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > > > at > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > > > at > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > > > at java.security.AccessController.doPrivileged(Native > Method) > > > at javax.security.auth.Subject.doAs(Subject.java:396) > > > at > org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > > > > > > at org.apache.hadoop.ipc.Client.call(Client.java:739) > > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > > > at $Proxy0.addBlock(Unknown Source) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > > at > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > ava > > :39) > > > at > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > orI > > mpl.j > > > ava:25) > > > at java.lang.reflect.Method.invoke(Method.java:597) > > > at > > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryIn > voc > > ation > > > Handler.java:82) > > > at > > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocati > onH > > andle > > > r.java:59) > > > at $Proxy0.addBlock(Unknown Source) > > > at > > > > > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock( > DFS > > Clien > > > t.java:2873) > > > ... 3 more > > > > > > 10/01/21 19:35:58 WARN hdfs.DFSClient: Error Recovery for block null > bad > > > datanode[0] nodes == null > > > 10/01/21 19:35:58 WARN hdfs.DFSClient: Could not get block locations. > Source > > > file "/user/root/input/xaa" - Aborting... > > > put: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The > quota of > > > /user/root/input is exceeded: namespace quota=-1 file count=2, > diskspace > > > quota=192000 diskspace=134217728 > > > > > > When I saw the above error messages,it was described as > diskspace=134217728. > > > Perhaps, I think that it is "134217728/1024^2=128MByte". > > > > > > I understood that this is why The quota(192000=192Kbyte) of > /user/root/input > > > was exceeded. > > > > > > Q1：Why does diskspace become MByet when the file of KByte is input?(1024 > > > calculations in HDFS) > > > > > > [r...@drbd-test-vm03 current]# ./bin/hadoop dfs -lsr > /user/root/input/xaa > > > -rw-r--r-- 2 root supergroup 0 2010-01-21 19:35 > > > /user/root/input/xaa > > > > > > Quota exceed exception,but it creates file of size 0. > > > > > > It seems that this is the similar problem as the problem described in > > > following URL. > > > http://issues.apache.org/jira/browse/HDFS-172 > > > > > > Has not this problem solved yet? > > > > > > Q2：Is there a person who has information for this problem that quota > exceed > > > exception,but it creates file of size 0? > > > > > > ★Since the second times > > > The result was different the first time and the second times. > > > It doesn't make an error even if it exceeds capacity since the second > times. > > > > > > Imput file's capacity is 64000=64k. > > > Reprication figure is 2. > > > Calculated result > > > 102400-(64000×2)=-25600<-- Why does it become a "-" mark without > becoming > > an > > > error? > > > > > > I thought that the space quota is a limit on the number of bytes used > by > > files > > > at that directory. > > > However, the result shows the thing to which my assumption is wrong. > > > > > > Q3：Why did input file's capacity exceed the limit(space quota)? > > > > > > [r...@drbd-test-vm03 current]# ./bin/hadoop dfsadmin -setSpaceQuota > 100K > > > /user/root/input/ > > > [r...@drbd-test-vm03 current]# ./bin/hadoop fs -count -q > /user/root/input/ > > > none inf 102400 102400 > > 1 > > > 0 0 hdfs://drbd-test-vm03/user/root/input > > > [r...@drbd-test-vm03 current]# ./bin/hadoop dfs -put xaa > /user/root/input/ > > > [r...@drbd-test-vm03 current]# ./bin/hadoop fs -count -q > /user/root/input/ > > > none inf 102400 -25600 > > 1 > > > 1 64000 hdfs://drbd-test-vm03/user/root/input > > > > > > Best regards, > > > Tadashi. > > >
