[
https://issues.apache.org/jira/browse/HDFS-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-877:
-----------------------------
Attachment: hdfs-877.txt
bq. Is it possible that pos + bytesToRead > totalBytesToRead in case that the
last byte to read does not align at the chunk boundary?
I was originally thinking that the DN sent a partial chunk (with computed
checksum) in that case but you're absolutely right. This new patch changes the
condition to >=. I also realized that I was neglecting to account for the
initial padding when computing this value. I fixed both problems and renamed
totalBytesToRead to bytesNeededToFinish.
Given that this behavior doesn't seem to be well covered by tests I think we
should also open another JIRA to directly test BlockReader and ensure that
CHECKSUM_OK gets sent in the various cases.
> Client-driven block verification not functioning
> ------------------------------------------------
>
> Key: HDFS-877
> URL: https://issues.apache.org/jira/browse/HDFS-877
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client, test
> Affects Versions: 0.20.1, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hdfs-877.txt, hdfs-877.txt, hdfs-877.txt
>
>
> This is actually the reason for HDFS-734 (TestDatanodeBlockScanner timing
> out). The issue is that DFSInputStream relies on readChunk being called one
> last time at the end of the file in order to receive the
> lastPacketInBlock=true packet from the DN. However, DFSInputStream.read
> checks pos < getFileLength() before issuing the read. Thus gotEOS never
> shifts to true and checksumOk() is never called.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
