[
https://issues.apache.org/jira/browse/HDFS-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518465#comment-14518465
]
Zhe Zhang commented on HDFS-7678:
---------------------------------
Thanks Andrew for the review; it's very helpful. Some quick feedback while I
work on the harder parts:
bq. waitNextCompletion, shouldn't the read timeout be an overall timeout
Great idea. Otherwise the timeout policy is too strict in the beginning and too
loose toward the end.
bq. throwing InterruptedException on empty futures
This part (also the main {{waitNextCompletion}} logic) was actually "inherited"
from {{DFSInputStream#getFirstToComplete}}. I think we should take care of this
issue together with other {{InterruptedException}} updates in the planed
follow-on JIRA (against trunk). I will update {{waitNextCompletion}} to get rid
of this {{InterruptedException}} under this JIRA.
bq. Do we actually need missingBlkIndices or the non-success cases? It's the
set complement of fetchedBlkIndices.
Not really. {{missingBlkIndices}} has all _confirmed_ missing blocks while
{{fetchedBlkIndices}} has all fetched blocks that _cover the max missing span_.
For example, if cell size is 4k, you want to read range 2k~8k, and block #1 is
missing, then {{missingBlkIndices}} should contain only _1_ and
{{fetchedBlkIndices}} is empty, since block #0 needs to be refetched (we only
have half of it for recovery).
bq. We always go through a function called "fetchExtraBlks..."
Good catch and I think that's what fails {{TestReadStripedFile}} and
{{TestDFSStripedInputStream}}
bq. I wonder if it'd be better to do all the fetching first (including parity
if necessary),
It's an appealing idea. [~hitliuyi] has an interesting logic of "inserting a
new Future when finding a failed Future" logic under HDFS-7348. I'll try to
leverage that.
> Erasure coding: DFSInputStream with decode functionality
> --------------------------------------------------------
>
> Key: HDFS-7678
> URL: https://issues.apache.org/jira/browse/HDFS-7678
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: HDFS-7285
> Reporter: Li Bo
> Assignee: Zhe Zhang
> Attachments: BlockGroupReader.patch, HDFS-7678-HDFS-7285.002.patch,
> HDFS-7678.000.patch, HDFS-7678.001.patch
>
>
> A block group reader will read data from BlockGroup no matter in striping
> layout or contiguous layout. The corrupt blocks can be known before
> reading(told by namenode), or just be found during reading. The block group
> reader needs to do decoding work when some blocks are found corrupt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
