[
https://issues.apache.org/jira/browse/HDFS-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-7678:
----------------------------
Attachment: HDFS-7678-HDFS-7285.010.patch
Thanks Jing for the review; it's very helpful. Updating the patch to address
the comments.
bq. futures and stripedReadsService can be converted into local variables
inside of DFSStripedInputStream.
Thanks for pointing it out. Fixed.
bq. fetchBlockByteRange needs to be split into multiple functions
Refactored. Let me know how the new structure looks.
bq. I have some concerns about the current timeout mechanism:
Very good point re. unfair timeout for parity blocks. I agree we should handle
slow readers (not as slow enough to trigger {{BlockReader}} timeout) in a
separate JIRA.
bq. Looks like the following code will not add r.index into fetchedBlkIndices
for blk_0 in the java comment example?
You are right; and this is the expected logic. {{fetchedBlkIndices}} only
contains blocks fetched at maximum portion (therefore not requiring any
re-fetch even if other blocks are missing). In the example, if blk_0 returns a
{{SUCCESS}}, it's not put into {{fetchedBlkIndices}}, then if blk_1 request
returns as {{FAILED}}, we will issue another request to fetch blk_0 with
maximum portion. At that time, if the re-fetch succeeds, blk_0 will enter
{{fetchedBlkIndices}}.
bq. When the reading length is smaller than a full stripe, this "if" will not
be hit.
That is right; and it is the expected logic. This part of code (separated out
as {{processReadTaskEvents}} in the new patch) uses an event-driven flow. If
all original read requests succeed, no recovery read future will be inserted,
so {{futures}} will just drain.
bq. We also need to check if the number of missing blocks is already > number
of parity blocks. In that case we should fail the read.
Good point! Fixed.
bq. We also need to check if the number of missing blocks is already > number
of parity blocks. In that case we should fail the read.
Totally agreed. In particular, we should add more seek-and-read tests and try
to emulate more timing-related scenarios. I believe [~xinwei] is working on
that under HDFS-8201 and HDFS-8202.
> Erasure coding: DFSInputStream with decode functionality (pread)
> ----------------------------------------------------------------
>
> Key: HDFS-7678
> URL: https://issues.apache.org/jira/browse/HDFS-7678
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: HDFS-7285
> Reporter: Li Bo
> Assignee: Zhe Zhang
> Attachments: BlockGroupReader.patch, HDFS-7678-HDFS-7285.002.patch,
> HDFS-7678-HDFS-7285.003.patch, HDFS-7678-HDFS-7285.004.patch,
> HDFS-7678-HDFS-7285.005.patch, HDFS-7678-HDFS-7285.006.patch,
> HDFS-7678-HDFS-7285.007.patch, HDFS-7678-HDFS-7285.008.patch,
> HDFS-7678-HDFS-7285.009.patch, HDFS-7678-HDFS-7285.010.patch,
> HDFS-7678.000.patch, HDFS-7678.001.patch
>
>
> A block group reader will read data from BlockGroup no matter in striping
> layout or contiguous layout. The corrupt blocks can be known before
> reading(told by namenode), or just be found during reading. The block group
> reader needs to do decoding work when some blocks are found corrupt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
