I noticed the same recently. For me it happened since the datanodes were
deleting lot of blocks. I was doing something like :
bin/hadoop fs -rm 4Gb; sleep 10; bin/hadoop fs -put 4Gb-input 4Gb;
This is because, when datanode is deleting blocks it does not inform the
namenode about the new blocks it received... I think it should not let
deletion block other activities like this.
Increasing the sleep to 30 removed it.
Raghu.
Christian Saar wrote:
Hi all,
When we are copying big files in our hadoop cluster, we become between 0
and 4 times the "NotReplicatedYetException".
Is there any Solution to prevent this Exception?
I thing it slows down?
Bigger Buffers or anything else?
Each Node has 3x 1Gb/s bonded interface.
Each Datanode has an 3Ware 9650 24 Port with 24 1TB Western Digital in
RAID 6 with one Hotspare.
all Systems are CentOS 5.1 64bit with 64bit Java JDK 1.6.0_06
08/07/09 15:28:17 WARN dfs.DFSClient: NotReplicatedYetException sleeping
/rbc/testadacor/5.dat retries left 1
08/07/09 15:28:21 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.dfs.NotReplicatedYetException: Not replicated
yet:/rbc/testadacor/5.dat
at
org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1131)
at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
at org.apache.hadoop.ipc.Client.call(Client.java:557)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
08/07/09 15:28:21 INFO dfs.DFSClient: Waiting for replication for 6 seconds
08/07/09 15:28:21 WARN dfs.DFSClient: NotReplicatedYetException sleeping
/rbc/testadacor/5.dat retries left 0
08/07/09 15:28:27 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.dfs.NotReplicatedYetException: Not replicated
yet:/rbc/testadacor/5.dat
at
org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1131)
at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
at org.apache.hadoop.ipc.Client.call(Client.java:557)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
08/07/09 15:28:27 INFO dfs.DFSClient: Waiting for replication for 12 seconds
08/07/09 15:28:27 WARN dfs.DFSClient: NotReplicatedYetException sleeping
/rbc/testadacor/5.dat retries left -1
