[
https://issues.apache.org/jira/browse/HDFS-10667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389290#comment-15389290
]
Yuanbo Liu commented on HDFS-10667:
-----------------------------------
[~yzhangal] I did background research about this jira and have read your
impressive conversation in HDFS-10587. This jira seems to be derived from
HDFS-10587, so I'm sorry for taking this jira if you intend to assign it to
someone else.
Here is my understanding about the offset in the file, offset in block, and
offset in packet:
{code}
The offset in the file: the sequence number of this packet in the pipeline
The offset in block: preserved in PacketHeader as offsetInBlock
The offset in packet: 81920 is the offset in packet as well as the position in
the exception
{code}
I've prepared a patch for this jira, but please feel free to take it over if
you have other plan.
> Report more accurate info about data corruption location
> --------------------------------------------------------
>
> Key: HDFS-10667
> URL: https://issues.apache.org/jira/browse/HDFS-10667
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, hdfs
> Reporter: Yongjun Zhang
> Assignee: Yuanbo Liu
>
> Per
> https://issues.apache.org/jira/browse/HDFS-10587?focusedCommentId=15376897&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15376897
> 129.77 report:
> {code}
> 2016-07-13 11:49:01,512 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Receiving blk_1116167880_42906656 src: /10.6.134.229:43844 dest:
> /10.6.129.77:5080
> 2016-07-13 11:49:01,543 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Checksum error in block blk_1116167880_42906656 from /10.6.134.229:43844
> org.apache.hadoop.fs.ChecksumException: Checksum error:
> DFSClient_NONMAPREDUCE_2019484565_1 at 81920 exp: 1352119728 got: -1012279895
> at
> org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native
> Method)
> at
> org.apache.hadoop.util.NativeCrc32.verifyChunkedSumsByteArray(NativeCrc32.java:69)
> at
> org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:347)
> at
> org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:294)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.verifyChunks(BlockReceiver.java:421)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:558)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:789)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:917)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:174)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:80)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:244)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-13 11:49:01,543 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exception for blk_1116167880_42906656
> java.io.IOException: Terminating due to a checksum error.java.io.IOException:
> Unexpected checksum mismatch while writing blk_1116167880_42906656 from
> /10.6.134.229:43844
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:571)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:789)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:917)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:174)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:80)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:244)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> and
> https://issues.apache.org/jira/browse/HDFS-10587?focusedCommentId=15378879&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15378879
> {quote}
> While verifying only packet, the position mentioned in the checksum
> exception, is relative to packet buffer offset, not the block offset. So
> 81920 is the offset in the exception.
> {quote}
> Create this jira to report more accurate corruption location information: the
> offset in the file, offset in block, and offset in packet.
> See
> https://issues.apache.org/jira/browse/HDFS-10587?focusedCommentId=15387083&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15387083
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
