Hi, Mithila, "File /user/mithila/test/20417.txt could only be replicated to 0 nodes, instead of 1"
I think your datanode isn't working properly. please take a look at log file of your datanode (logs/*datanode*.log). If there is no error in that log file, I've heard that hadoop can sometimes mark a datanode as "BAD" and refuses to send the block to that node, this can be the cause. (List, please correct me if I'm wrong!) Hope this helps, Rasit 2009/2/6 Mithila Nagendra <[email protected]>: > Hey all > I was trying to run the word count example on one of the hadoop systems I > installed, but when i try to copy the text files from the local file system > to the DFS, it throws up the following exception: > > [mith...@node02 hadoop]$ jps > 8711 JobTracker > 8805 TaskTracker > 8901 Jps > 8419 NameNode > 8642 SecondaryNameNode > [mith...@node02 hadoop]$ cd .. > [mith...@node02 mithila]$ ls > hadoop hadoop-0.17.2.1.tar hadoop-datastore test > [mith...@node02 mithila]$ hadoop/bin/hadoop dfs -copyFromLocal test test > 09/02/06 11:26:26 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: > java.io.IOException: File /user/mithila/test/20417.txt could only be > replicated to 0 nodes, instead of 1 > at > org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145) > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) > > at org.apache.hadoop.ipc.Client.call(Client.java:557) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2335) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2220) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842) > > 09/02/06 11:26:26 WARN dfs.DFSClient: NotReplicatedYetException sleeping > /user/mithila/test/20417.txt retries left 4 > 09/02/06 11:26:27 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: > java.io.IOException: File /user/mithila/test/20417.txt could only be > replicated to 0 nodes, instead of 1 > at > org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145) > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) > > at org.apache.hadoop.ipc.Client.call(Client.java:557) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2335) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2220) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842) > > 09/02/06 11:26:27 WARN dfs.DFSClient: NotReplicatedYetException sleeping > /user/mithila/test/20417.txt retries left 3 > 09/02/06 11:26:28 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: > java.io.IOException: File /user/mithila/test/20417.txt could only be > replicated to 0 nodes, instead of 1 > at > org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145) > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) > > at org.apache.hadoop.ipc.Client.call(Client.java:557) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2335) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2220) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842) > > 09/02/06 11:26:28 WARN dfs.DFSClient: NotReplicatedYetException sleeping > /user/mithila/test/20417.txt retries left 2 > 09/02/06 11:26:29 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: > java.io.IOException: File /user/mithila/test/20417.txt could only be > replicated to 0 nodes, instead of 1 > at > org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145) > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) > > at org.apache.hadoop.ipc.Client.call(Client.java:557) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2335) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2220) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842) > > 09/02/06 11:26:29 WARN dfs.DFSClient: NotReplicatedYetException sleeping > /user/mithila/test/20417.txt retries left 1 > 09/02/06 11:26:32 WARN dfs.DFSClient: DataStreamer Exception: > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File > /user/mithila/test/20417.txt could only be replicated to 0 nodes, instead of > 1 > at > org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145) > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) > > at org.apache.hadoop.ipc.Client.call(Client.java:557) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2335) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2220) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842) > > 09/02/06 11:26:32 WARN dfs.DFSClient: Error Recovery for block null bad > datanode[0] > copyFromLocal: Could not get block locations. Aborting... > Exception closing file /user/mithila/test/20417.txt > java.io.IOException: Could not get block locations. Aborting... > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2081) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1300(DFSClient.java:1702) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1818) > > Does anyone have an idea as to what could be wrong! Im fresh outta ideas! > > Thanks! > Mithila > -- M. Raşit ÖZDAŞ
