[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506131#comment-14506131
]
Yi Liu edited comment on HDFS-8033 at 4/22/15 12:42 AM:
--------------------------------------------------------
Thanks Zhe for the response, and also thanks Jing for the review and good
comments!
Yes, it's a good idea to maintain a list of current DataNodes.
{quote}
DFSStripedInputStream#readBuffer does switch the blockReader. So after reading
cell_0, we'll switch to the next blockReader and read cell_1.
{quote}
{{DFSStripedInputStream#readBuffer}} can switch the blockReader, that's true,
but it depends on the caller to call, currently for one stateful read,
{{DFSInputStream#readWithStrategy}} only call it one time, so the result only
contains part of the expected data.
I didn't see you do loop (parallelly) all blockReaders for the striped block in
{{DFSStripedInputStream#readBuffer}}.
I have another comments,
I think we need to change {{closeCurrentBlockReader}} to
{{closeCurrentBlockReaders}} in {{DFSInputStream}} and override it in
{{DFSStripedInputStream}}, otherwise there is leak for {{blockReaders}}.
was (Author: hitliuyi):
Thanks Zhe for the response, and also thanks Jing for the review and good
comments!
Yes, it's a good idea to maintain a list of current DataNodes.
{quote}
DFSStripedInputStream#readBuffer does switch the blockReader. So after reading
cell_0, we'll switch to the next blockReader and read cell_1.
{quote}
{{DFSStripedInputStream#readBuffer}} can switch the blockReader, that's true,
but it depends on the caller to call, currently for one stateful read,
{{DFSInputStream#readWithStrategy}} only call it one time, so the result only
contains part of the expected data.
I didn't see you do loop (parallelly) all blockReaders for the striped block in
{{DFSStripedInputStream#readBuffer}}.
I have another comments,
I think we need to change {{closeCurrentBlockReader}} to
{{closeCurrentBlockReaders}} in {{DFSInputStream}} and overwrite it in
{{DFSStripedInputStream}}, otherwise there is leak for {{blockReaders}}.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---------------------------------------------------------------------------
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Zhe Zhang
> Assignee: Zhe Zhang
> Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
