[
https://issues.apache.org/jira/browse/HDFS-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250020#comment-13250020
]
Uma Maheswara Rao G commented on HDFS-3222:
-------------------------------------------
Thanks a lot, Todd for your time.
{quote}
Yea – treat it the same as we treat a corrupt file.
{quote}
It may be difficult for the clients to differ whether this is real corruption
or it will be recovered after DN reports to NN.
{quote}
Alternatively, maybe this is so rare that it doesn't matter, and it's OK to
disallow reading from an unrecovered file whose last block is missing all of
its block locations after a restart.
{quote}
Let's take others opinion also on this?
Others also, please comment on this point.
{quote}
That's true. Is it possible for us to change the client code to defer this code
path until either (a) the client wants to read from the partial block, or (b)
the client explictly asks for the file length?
{quote}
you mean reader will pass the option? (a) or (b).
> DFSInputStream#openInfo should not silently get the length as 0 when
> locations length is zero for last partial block.
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3222
> URL: https://issues.apache.org/jira/browse/HDFS-3222
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 1.0.3, 2.0.0, 3.0.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-3222-Test.patch
>
>
> I have seen one situation with Hbase cluster.
> Scenario is as follows:
> 1)1.5 blocks has been written and synced.
> 2)Suddenly cluster has been restarted.
> Reader opened the file and trying to get the length., By this time partial
> block contained DNs are not reported to NN. So, locations for this partial
> block would be 0. In this case, DFSInputStream assumes that, 1 block size as
> final size.
> But reader also assuming that, 1 block size is the final length and setting
> his end marker. Finally reader ending up reading only partial data. Due to
> this, HMaster could not replay the complete edits.
> Actually this happend with 20 version. Looking at the code, same should
> present in trunk as well.
> {code}
> int replicaNotFoundCount = locatedblock.getLocations().length;
>
> for(DatanodeInfo datanode : locatedblock.getLocations()) {
> ..........
> ..........
> // Namenode told us about these locations, but none know about the replica
> // means that we hit the race between pipeline creation start and end.
> // we require all 3 because some other exception could have happened
> // on a DN that has it. we want to report that error
> if (replicaNotFoundCount == 0) {
> return 0;
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
