[
https://issues.apache.org/jira/browse/HDFS-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-877:
-----------------------------
Description:
This is actually the reason for HDFS-734 (TestDatanodeBlockScanner timing out).
The issue is that DFSInputStream relies on readChunk being called one last time
at the end of the file in order to receive the lastPacketInBlock=true packet
from the DN. However, DFSInputStream.read checks pos < getFileLength() before
issuing the read. Thus gotEOS never shifts to true and checksumOk() is never
called.
was:This is actually the reason for HDFS-734 (TestDatanodeBlockScanner timing
out). The issue is that DFSInputStream relies on readChunk being called one
last time at the end of the file in order to receive the lastPacketInBlock=true
packet from the DN. However, DFSInputStream.read checks pos < getFileLength()
before issuing the read. Thus gotEOS never shifts to true and checksumOk() is
never called.
Priority: Critical (was: Major)
Summary: Client-driven block verification not functioning (was:
Client-driven checksum verification not functioning)
Upgrading this to blocker since it's a cause for a test failure.
Worth noting that this bug only affects the proactive "checksum OK" marking
that the client does after reading an entire block (thus avoiding the periodic
scan on the DN). If the checksum is found to be invalid on the client, it still
reports the bad block to the NN just fine. So, this isn't a dataloss bug, it's
just a broken optimization and a failing test.
> Client-driven block verification not functioning
> ------------------------------------------------
>
> Key: HDFS-877
> URL: https://issues.apache.org/jira/browse/HDFS-877
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0, 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
>
> This is actually the reason for HDFS-734 (TestDatanodeBlockScanner timing
> out). The issue is that DFSInputStream relies on readChunk being called one
> last time at the end of the file in order to receive the
> lastPacketInBlock=true packet from the DN. However, DFSInputStream.read
> checks pos < getFileLength() before issuing the read. Thus gotEOS never
> shifts to true and checksumOk() is never called.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
