Hmm... There's no physical limitation, is there an artificial setting?
On Tue, Apr 12, 2011 at 4:27 PM, Jean-Daniel Cryans <[email protected]> wrote: > It says: > > 2011-04-12 16:16:17,157 DEBUG [IPC Server handler 7 on 51372] > namenode.ReplicationTargetChooser(408): Node > /default-rack/127.0.0.1:22967 is not chosen because the node does not > have enough space > > J-D > > On Tue, Apr 12, 2011 at 4:24 PM, Jason Rutherglen > <[email protected]> wrote: >> Ah, I had changed conf/log4j.properties. So I changed >> src/test/resources/log4j.properties, and now the -output file's much >> more verbose. I'm not sure I understand what's going on however. >> >> I'll try to make sense out of the log: >> >> http://pastebin.com/MrQJcbJr >> >> On Tue, Apr 12, 2011 at 3:38 PM, Stack <[email protected]> wrote: >>> You changed the src/test/resources/log4j.properties? >>> >>> Not sure why changing the block size would make a difference, why it >>> would even care. >>> >>> St.Ack >>> >>> On Tue, Apr 12, 2011 at 2:38 PM, Jason Rutherglen >>> <[email protected]> wrote: >>>> Thanks, I'm only seeing the error when I change the block size, either >>>> via DFSClient.create or via the Configuration dfs.block.size property. >>>> >>>> When I changed the log4j.properties to >>>> 'log4j.logger.org.apache.hadoop=WARN' I'm not seeing anything >>>> additional in the output in the target/surefire-reports directory. >>>> >>>> On Tue, Apr 12, 2011 at 12:59 PM, Gary Helmling <[email protected]> >>>> wrote: >>>>> Depends what the log4j.properties file that your code is picking up says. >>>>> mvn test or IDE "run" test classes should pick up >>>>> src/test/resources/log4j.properties, which will log to stderr. If that's >>>>> how you're running you could tweak the hadoop logging level to see if it >>>>> shows anything more. Change: >>>>> >>>>> log4j.logger.org.apache.hadoop=WARN >>>>> >>>>> to INFO or DEBUG. >>>>> >>>>> Also, mvn test will redirect the log output to >>>>> target/surefire-reports/org.apache.hadoop.hbase....-output.txt >>>>> >>>>> >>>>> >>>>> On Tue, Apr 12, 2011 at 12:43 PM, Jason Rutherglen < >>>>> [email protected]> wrote: >>>>> >>>>>> Where does MiniDFSCluster store the logs? I don't see a location, >>>>>> assuming it's different than stdout/err. >>>>>> >>>>>> On Tue, Apr 12, 2011 at 11:26 AM, Stack <[email protected]> wrote: >>>>>> > The datanodes are not starting? Anything about that in the log? >>>>>> > St.Ack >>>>>> > >>>>>> > On Tue, Apr 12, 2011 at 11:13 AM, Jason Rutherglen >>>>>> > <[email protected]> wrote: >>>>>> >> I'm running into an error when setting the DFS block size to be larger >>>>>> >> than the default. The following code is used to create the test >>>>>> >> cluster: >>>>>> >> >>>>>> >> Configuration conf = new Configuration(); >>>>>> >> MiniDFSCluster cluster = new MiniDFSCluster(conf, 2, true, null); >>>>>> >> FileSystem fileSys = cluster.getFileSystem(); >>>>>> >> >>>>>> >> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File >>>>>> >> /index/_0_0.tib could only be replicated to 0 nodes, instead of 1 >>>>>> >> at >>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1363) >>>>>> >> at >>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:449) >>>>>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>> >> at >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>> >> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>> >> at java.lang.reflect.Method.invoke(Method.java:616) >>>>>> >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) >>>>>> >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:961) >>>>>> >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:957) >>>>>> >> at java.security.AccessController.doPrivileged(Native Method) >>>>>> >> at javax.security.auth.Subject.doAs(Subject.java:416) >>>>>> >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:955) >>>>>> >> >>>>>> >> at org.apache.hadoop.ipc.Client.call(Client.java:740) >>>>>> >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) >>>>>> >> at $Proxy4.addBlock(Unknown Source) >>>>>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>> >> at >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>> >> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>> >> at java.lang.reflect.Method.invoke(Method.java:616) >>>>>> >> at >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) >>>>>> >> at >>>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) >>>>>> >> at $Proxy4.addBlock(Unknown Source) >>>>>> >> at >>>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3184) >>>>>> >> at >>>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3058) >>>>>> >> at >>>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2276) >>>>>> >> at >>>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2511) >>>>>> >> >>>>>> > >>>>>> >>>>> >>>> >>> >> >
