[
https://issues.apache.org/jira/browse/HDFS-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-877:
-----------------------------
Attachment: hdfs-877.txt
I took the following route to fixing this:
- BlockReader now knows how many bytes it's expected to read.
- After it has read this many bytes, it reads ahead one packet, expecting an
empty "end of stream" packet from the DN. In the case that this doesn't come
(or it's not the special end of stream packet) it throws IOE. If it does get
it, it sets gotEOS.
This current patch is a little ugly since it duplicates the
packet-header-reading logic. I have another version that depends on HDFS-881 to
clean that up, if 881 gets committed in the meantime.
> Client-driven block verification not functioning
> ------------------------------------------------
>
> Key: HDFS-877
> URL: https://issues.apache.org/jira/browse/HDFS-877
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client, test
> Affects Versions: 0.20.1, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-877.txt
>
>
> This is actually the reason for HDFS-734 (TestDatanodeBlockScanner timing
> out). The issue is that DFSInputStream relies on readChunk being called one
> last time at the end of the file in order to receive the
> lastPacketInBlock=true packet from the DN. However, DFSInputStream.read
> checks pos < getFileLength() before issuing the read. Thus gotEOS never
> shifts to true and checksumOk() is never called.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
