Iván,
Whether this was expected or an error depends on what happened on the
client. This could happen and would not be a bug if client was killed
for some other reason for e.g. But if client is also similarly surprised
then its a different case.
You could grep for this block in NameNode log and client. If you are
still interested in looking into this, I would suggest opening a jira.
Raghu.
Iván de Prado wrote:
Thanks,
I have tried with the trunk version and now the exception "Trying to
change block file offset of block blk_... to ... but actual size of file
is ..." has disappeared and the jobs don't seems to get blocked.
But I have another "Broken Pipe" and "EOF" exceptions in the dfs logs.
They seems similar to https://issues.apache.org/jira/browse/HADOOP-2042
ticket. The Jobs ends but not sure if they are executed smoothly. are
these exceptions normal? As example, the exceptions for the block
(6801211507359331627) appears in two nodes (I have 2 as replication) and
looks like:
hn2: 2008-03-31 05:03:13,736 INFO org.apache.hadoop.dfs.DataNode:
Datanode 0 forwarding connect ack to upstream firstbadlink is
hn2: 2008-03-31 05:03:14,507 INFO org.apache.hadoop.dfs.DataNode:
Receiving block blk_6801211507359331627 src: /172.16.3.6:38218
dest: /172.16.3.6:50010
hn2: 2008-03-31 05:04:14,528 INFO org.apache.hadoop.dfs.DataNode:
Exception in receiveBlock for block blk_6801211507359331627
java.io.EOFException
hn2: 2008-03-31 05:04:14,528 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 0 for block blk_6801211507359331627 Interrupted.
hn2: 2008-03-31 05:04:14,528 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 0 for block blk_6801211507359331627 terminating
hn2: 2008-03-31 05:04:14,530 INFO org.apache.hadoop.dfs.DataNode:
writeBlock blk_6801211507359331627 received exception
java.io.EOFException
hn2: 2008-03-31 05:04:14,530 ERROR org.apache.hadoop.dfs.DataNode:
172.16.3.4:50010:DataXceiver: java.io.EOFException
hn2: at java.io.DataInputStream.readInt(DataInputStream.java:375)
hn2: at org.apache.hadoop.dfs.DataNode
$BlockReceiver.receiveBlock(DataNode.java:2243)
hn2: at org.apache.hadoop.dfs.DataNode
$DataXceiver.writeBlock(DataNode.java:1157)
hn2: at org.apache.hadoop.dfs.DataNode
$DataXceiver.run(DataNode.java:938)
hn2: at java.lang.Thread.run(Thread.java:619)
hn4: 2008-03-31 05:03:13,590 INFO org.apache.hadoop.dfs.DataNode:
Datanode 0 forwarding connect ack to upstream firstbadlink is
hn4: 2008-03-31 05:03:14,506 INFO org.apache.hadoop.dfs.DataNode:
Receiving block blk_6801211507359331627 src: /172.16.3.6:41112
dest: /172.16.3.6:50010
hn4: 2008-03-31 05:03:26,825 INFO org.apache.hadoop.dfs.DataNode:
Exception in receiveBlock for block blk_6801211507359331627
java.io.EOFException
hn4: 2008-03-31 05:04:14,524 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder blk_6801211507359331627 1 Exception
java.net.SocketException: Broken pipe
hn4: at java.net.SocketOutputStream.socketWrite0(Native Method)
hn4: at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
hn4: at
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
hn4: at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
hn4: at org.apache.hadoop.dfs.DataNode
$PacketResponder.run(DataNode.java:1825)
hn4: at java.lang.Thread.run(Thread.java:619)
hn4:
hn4: 2008-03-31 05:04:14,525 INFO org.apache.hadoop.dfs.DataNode:
PacketResponder 1 for block blk_6801211507359331627 terminating
hn4: 2008-03-31 05:04:14,525 INFO org.apache.hadoop.dfs.DataNode:
writeBlock blk_6801211507359331627 received exception
java.io.EOFException
hn4: 2008-03-31 05:04:14,526 ERROR org.apache.hadoop.dfs.DataNode:
172.16.3.6:50010:DataXceiver: java.io.EOFException
hn4: at java.io.DataInputStream.readInt(DataInputStream.java:375)
hn4: at org.apache.hadoop.dfs.DataNode
$BlockReceiver.receiveBlock(DataNode.java:2243)
hn4: at org.apache.hadoop.dfs.DataNode
$DataXceiver.writeBlock(DataNode.java:1157)
hn4: at org.apache.hadoop.dfs.DataNode
$DataXceiver.run(DataNode.java:938)
hn4: at java.lang.Thread.run(Thread.java:619)
hn4:
Many thanks,
Iván de Prado Alonso
http://ivandeprado.blogspot.com/
