Hi all,
We are facing issues while porting some logs to HDFS. The way we are doing it
is using a simple java code which tries to read the file and writes to HDFS
using OutputStream. It was working perfectly fine and recently, we are getting
below error messages once in a while and when we try to read that data for that
data we are getting error message as "Could not obtain block" and the jobs are
failing.
Can some one tell me what would be the issue?
The error message while writing to HDFS is:
WARN dfs.DFSClient: DFSOutputStream ResponseProcessor
exception for block blk_-1005367228931083977_10012402java.io.IOException:
Bad response 1 for block blk_-1005367228931083977_10012402 from datanode
xxx.xxx.xxx.88:50010
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSCli
ent.java:2076)
09/06/08 14:26:41 WARN dfs.DFSClient: Error Recovery for block
blk_-1005367228931083977_10012402 bad datanode[1] xxx.xxx.xxx.88:50010
09/06/08 14:26:41 WARN dfs.DFSClient: Error Recovery for block
blk_-1005367228931083977_10012402 in pipeline xxx.xxx.xxx.79:50010,
xxx.xxx.xxx.88:50010, xxx.xxx.xxx.68:50010: bad datanode
xxx.xxx.xxx.88:50010
09/06/08 14:26:42 WARN dfs.DFSClient: DFSOutputStream ResponseProcessor
exception for block blk_-1005367228931083977_10012456java.io.IOException:
Bad response 1 for block blk_-1005367228931083977_10012456 from datanode
xxx.xxx.xxx.68:50010
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSCli
ent.java:2076)
09/06/08 14:26:42 WARN dfs.DFSClient: Error Recovery for block
blk_-1005367228931083977_10012456 bad datanode[1] xxx.xxx.xxx.68:50010
09/06/08 14:26:42 WARN dfs.DFSClient: Error Recovery for block
blk_-1005367228931083977_10012456 in pipeline xxx.xxx.xxx.79:50010,
xxx.xxx.xxx.68:50010: bad datanode xxx.xxx.xxx.68:50010
09/06/08 14:26:43 WARN dfs.DFSClient: DFSOutputStream ResponseProcessor
exception for block blk_-1005367228931083977_10012457java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSCli
ent.java:2052)
09/06/08 14:26:43 WARN dfs.DFSClient: Error Recovery for block
blk_-1005367228931083977_10012457 bad datanode[0] xxx.xxx.xxx.79:50010
IOException - java.io.IOException: All datanodes xxx.xxx.xxx.79:50010 are
bad. Aborting... while writing -
Thanks
Pallavi
