Does anyone have an expected or experienced write speed to HDFS outside
of Map/Reduce? Any recommendations on properties to tweak in
hadoop-site.xml?
Currently I have a multi-threaded writer where each thread is writing to
a different file. But after a while I get this:
java.io.IOException: Could not get block locations. Aborting...
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFS
Client.java:2081)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1300(DFSClient.ja
va:1702)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClie
nt.java:1818)
Which is perhaps indicating that the namenode is overwhelmed?
Thanks,
-Xavier
