Hi Boris,
Thank you for your reply.
> From your example it looks like you have namespace quota (not diskspace),
> i.e. quota for quantities of files (not size)
I used following command.
$ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
I thought this was a limit on total size(192000) of all the files
under the directory tree(/user/root/input/).
Does quota for quantities of files (not size) mean Name Quotas?
I found the following sentences.
Name Quotas is a hard limit on the number of file and directory.
Space Quotas is a hard limit on the number of bytes used by files.
Sorry,I'm confused about the difference between namespace and diskspace.
> Yes, I think this is by design. When HDFS creates a file, it doesn't know
> how big the file will be. So diskspace quota is checked on writing to the
> file (not on creation).
I understood that.
However, I think that the system recognize that the file exists even by 0
bytes.
Therefore, should I take the action deleted on the application side when it
becomes 0 bytes?
> What is your block size. I don't think it make sense to put quota for less
> then a blocksize.
My block size is 64MByte(default setting).
I understood that I should have set the quota larger than the block size.
File size is 134217728/1024^2=128MByte.(larger than the block size(64MByte))
Quota size is 256MByte(256*1024^2=268435456).(larger than the block
size(64MByte))
Number of replication is 2.
# ls -ltr xaa
-rw-r--r-- 1 root root 134217728 Jan 26 01:45 xaa
# ls -ltr xab
-rw-r--r-- 1 root root 134217728 Jan 26 01:45 xab
# ./bin/hadoop dfsadmin -setSpaceQuota 256m /user/root/input/
# ./bin/hadoop fs -count -q /user/root/input
none inf 268435456 268435456 1
0 0 hdfs://drbd-test-vm03/user/root/input
# ./bin/hadoop dfs -put xaa /user/root/input/
# ./bin/hadoop fs -count -q /user/root/input
none inf 268435456 0 1
1 134217728 hdfs://drbd-test-vm03/user/root/input
# ./bin/hadoop dfs -put xab /user/root/input/
# ./bin/hadoop fs -count -q /user/root/input
none inf 268435456 -268435456 1
2 268435456 hdfs://drbd-test-vm03/user/root/input
I thought that it became an error when the quota size was exceeded.
However, it was a "-" mark without becoming an error.
Why didn't the Space Quota setting(hard limit on the number of bytes) become
effective?
Best regards,
Tadashi.
> -----Original Message-----
> From: Boris Shkolnik [mailto:bo...@yahoo-inc.com]
> Sent: Tuesday, January 26, 2010 5:20 AM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: HDFS Quota
>
> >>> /user/root/input is exceeded: namespace quota=-1 file count=2,
> From your example it looks like you have namespace quota (not diskspace),
> i.e. quota for quantities of files (not size)
>
>
> > Quota exceed exception,but it creates file of size 0.
> Yes, I think this is by design. When HDFS creates a file, it doesn't know
> how big the file will be. So diskspace quota is checked on writing to the
> file (not on creation).
>
> >Q3：Why did input file's capacity exceed the limit(space quota)?
> What is your block size. I don't think it make sense to put quota for less
> then a blocksize.
>
> Boris.
>
> On 1/25/10 1:01 AM, "tate...@nttdata.co.jp" <tate...@nttdata.co.jp> wrote:
>
> > Hi everyone,
> >
> > I set Space Quotas for the amount of space on HDFS.
> > But I have some questions.
> >
> > ■Question
> > Q1：Why does diskspace become MByet when the file of KByte is input?(1024
> > calculations in HDFS)
> > Q2：Is there a person who has information for this problem that quota exceed
> > exception,but it creates file of size 0?
> > Q3：Why did input file's capacity exceed the limit(space quota)?
> >
> >
> > ■Details of question
> > ★First time
> > Even when I put file which was smaller than the capacity set in Space Quota,
> > it became an error.
> >
> > The input file's capacity which I put is 64000=64kbyte.
> >
> > ・input file
> > $ ls -ltr
> > $ -rw-r--r-- 1 root root 64000 Jan 21 04:42 xaa
> > $ du -h xaa
> > 68K xaa
> >
> > ・hdfs-site.xml(replication)
> > <property>
> > <name>dfs.replication</name>
> > <value>2</value>
> > </property>
> >
> > ・Space Quota
> > $ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
> > $ ./bin/hadoop fs -count -q /user/root/input/
> > none inf 192000 192000
> 1
> > 0 0 hdfs://drbd-test-vm03/user/root/input
> > $ ./bin/hadoop dfs -put input/xaa /user/root/input/
> > 10/01/21 19:35:58 WARN hdfs.DFSClient: DataStreamer Exception:
> > org.apache.hadoop.hdfs.protocol.QuotaExceededException:
> > org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> > /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
> > quota=192000 diskspace=134217728
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> > at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAc
> cesso
> > rImpl.java:39)
> > at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConst
> ructo
> > rAccessorImpl.java:27)
> > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > at
> >
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteExceptio
> n.jav
> > a:96)
> > at
> >
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteExcepti
> on.ja
> > va:58)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFS
> Clien
> > t.java:2875)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DF
> SClie
> > nt.java:2755)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.ja
> va:20
> > 46)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClie
> nt.ja
> > va:2232)
> > Caused by: org.apache.hadoop.ipc.RemoteException:
> > org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> > /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
> > quota=192000 diskspace=134217728
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuot
> a(INo
> > deDirectoryWithQuota.java:161)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateNumI
> temsI
> > nTree(INodeDirectoryWithQuota.java:134)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirector
> y.jav
> > a:859)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.j
> ava:2
> > 65)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNames
> ystem
> > .java:1427)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FS
> Names
> > ystem.java:1274)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:42
> 2)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
> :39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
> mpl.j
> > ava:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:739)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > at $Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
> :39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
> mpl.j
> > ava:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvoc
> ation
> > Handler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationH
> andle
> > r.java:59)
> > at $Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFS
> Clien
> > t.java:2873)
> > ... 3 more
> >
> > 10/01/21 19:35:58 WARN hdfs.DFSClient: Error Recovery for block null bad
> > datanode[0] nodes == null
> > 10/01/21 19:35:58 WARN hdfs.DFSClient: Could not get block locations. Source
> > file "/user/root/input/xaa" - Aborting...
> > put: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> > /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
> > quota=192000 diskspace=134217728
> >
> > When I saw the above error messages,it was described as diskspace=134217728.
> > Perhaps, I think that it is "134217728/1024^2=128MByte".
> >
> > I understood that this is why The quota(192000=192Kbyte) of /user/root/input
> > was exceeded.
> >
> > Q1：Why does diskspace become MByet when the file of KByte is input?(1024
> > calculations in HDFS)
> >
> > [r...@drbd-test-vm03 current]# ./bin/hadoop dfs -lsr /user/root/input/xaa
> > -rw-r--r-- 2 root supergroup 0 2010-01-21 19:35
> > /user/root/input/xaa
> >
> > Quota exceed exception,but it creates file of size 0.
> >
> > It seems that this is the similar problem as the problem described in
> > following URL.
> > http://issues.apache.org/jira/browse/HDFS-172
> >
> > Has not this problem solved yet?
> >
> > Q2：Is there a person who has information for this problem that quota exceed
> > exception,but it creates file of size 0?
> >
> > ★Since the second times
> > The result was different the first time and the second times.
> > It doesn't make an error even if it exceeds capacity since the second times.
> >
> > Imput file's capacity is 64000=64k.
> > Reprication figure is 2.
> > Calculated result
> > 102400-(64000×2)=-25600<-- Why does it become a "-" mark without becoming
> an
> > error?
> >
> > I thought that the space quota is a limit on the number of bytes used by
> files
> > at that directory.
> > However, the result shows the thing to which my assumption is wrong.
> >
> > Q3：Why did input file's capacity exceed the limit(space quota)?
> >
> > [r...@drbd-test-vm03 current]# ./bin/hadoop dfsadmin -setSpaceQuota 100K
> > /user/root/input/
> > [r...@drbd-test-vm03 current]# ./bin/hadoop fs -count -q /user/root/input/
> > none inf 102400 102400
> 1
> > 0 0 hdfs://drbd-test-vm03/user/root/input
> > [r...@drbd-test-vm03 current]# ./bin/hadoop dfs -put xaa /user/root/input/
> > [r...@drbd-test-vm03 current]# ./bin/hadoop fs -count -q /user/root/input/
> > none inf 102400 -25600
> 1
> > 1 64000 hdfs://drbd-test-vm03/user/root/input
> >
> > Best regards,
> > Tadashi.
> >
