[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938676#comment-14938676 ] Hudson commented on HDFS-8033: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/]) HDFS-8033. Erasure coding: stateful (non-positional) read from files in (zhezhang: rev 89d33785780f98a58e1e81eca2c27165840475df) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: ByteBufferStrategy_support_len.patch, > HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, > HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch, > HDFS-8033.006.patch, HDFS-8033.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938484#comment-14938484 ] Hudson commented on HDFS-8033: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/]) HDFS-8033. Erasure coding: stateful (non-positional) read from files in (zhezhang: rev 89d33785780f98a58e1e81eca2c27165840475df) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: ByteBufferStrategy_support_len.patch, > HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, > HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch, > HDFS-8033.006.patch, HDFS-8033.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938107#comment-14938107 ] Hudson commented on HDFS-8033: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/]) HDFS-8033. Erasure coding: stateful (non-positional) read from files in (zhezhang: rev 89d33785780f98a58e1e81eca2c27165840475df) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: ByteBufferStrategy_support_len.patch, > HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, > HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch, > HDFS-8033.006.patch, HDFS-8033.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937887#comment-14937887 ] Hudson commented on HDFS-8033: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/]) HDFS-8033. Erasure coding: stateful (non-positional) read from files in (zhezhang: rev 89d33785780f98a58e1e81eca2c27165840475df) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: ByteBufferStrategy_support_len.patch, > HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, > HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch, > HDFS-8033.006.patch, HDFS-8033.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937633#comment-14937633 ] Hudson commented on HDFS-8033: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/]) HDFS-8033. Erasure coding: stateful (non-positional) read from files in (zhezhang: rev 89d33785780f98a58e1e81eca2c27165840475df) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: ByteBufferStrategy_support_len.patch, > HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, > HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch, > HDFS-8033.006.patch, HDFS-8033.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937384#comment-14937384 ] Hudson commented on HDFS-8033: -- FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8548/]) HDFS-8033. Erasure coding: stateful (non-positional) read from files in (zhezhang: rev 89d33785780f98a58e1e81eca2c27165840475df) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: ByteBufferStrategy_support_len.patch, > HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, > HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch, > HDFS-8033.006.patch, HDFS-8033.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937187#comment-14937187 ] Hudson commented on HDFS-8033: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/]) HDFS-8033. Erasure coding: stateful (non-positional) read from files in (zhezhang: rev 89d33785780f98a58e1e81eca2c27165840475df) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFile.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Fix For: HDFS-7285 > > Attachments: ByteBufferStrategy_support_len.patch, > HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, > HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch, > HDFS-8033.006.patch, HDFS-8033.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512271#comment-14512271 ] Zhe Zhang commented on HDFS-8033: - Thanks for the reviews from Jing, Walter, and Yi! I just committed the patch. > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: ByteBufferStrategy_support_len.patch, > HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, > HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch, > HDFS-8033.006.patch, HDFS-8033.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512254#comment-14512254 ] Jing Zhao commented on HDFS-8033: - Thanks Zhe! +1 for the 007 patch. > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: ByteBufferStrategy_support_len.patch, > HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, > HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch, > HDFS-8033.006.patch, HDFS-8033.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512010#comment-14512010
]
Jing Zhao commented on HDFS-8033:
-
Yeah, in fact I think we do not need to extend TestDFSStripedInputStream. We
only need to read the file three times in
{{testOneFileUsingDFSStripedInputStream}}: pread, stateful read with byte[],
and stateful read with bytebuffer. But this can be done in a separate jira.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: ByteBufferStrategy_support_len.patch,
> HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch,
> HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch,
> HDFS-8033.006.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512003#comment-14512003
]
Zhe Zhang commented on HDFS-8033:
-
Thanks Jing for reviewing and the feedback on testing!
If we add ByteBuffer testing that will be yet another subclass of
{{TestDFSStripedInputStream}}. Maybe we should leave it until HDFS-8201 does
some refactoring and cleanup. In particular, we should try to avoid setting up
cluster and writing file too many times. Does it sound OK to you?
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: ByteBufferStrategy_support_len.patch,
> HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch,
> HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch,
> HDFS-8033.006.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511973#comment-14511973 ] Jing Zhao commented on HDFS-8033: - Thanks Zhe! The 006 patch looks pretty good to me. +1 > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: ByteBufferStrategy_support_len.patch, > HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, > HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch, > HDFS-8033.006.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511959#comment-14511959
]
Zhe Zhang commented on HDFS-8033:
-
Thanks for Jing's comment. I just noticed {{TestDFSStripedInputStream}}; later
we should merge or refactor the 2 test classes (maybe in HDFS-8201). It's a
good idea to extend that test. 006 patch takes care of this.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: ByteBufferStrategy_support_len.patch,
> HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch,
> HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch,
> HDFS-8033.006.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511881#comment-14511881
]
Jing Zhao commented on HDFS-8033:
-
Thanks for updating the patch, Zhe! A very quick comment is that let's extend
{{TestDFSStripedInputStream#testOneFileUsingDFSStripedInputStream}} to test
stateful read (with either byte[] or ByteBuffer input).
I will review other part of the patch and post comments later today.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: ByteBufferStrategy_support_len.patch,
> HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch,
> HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511833#comment-14511833 ] Zhe Zhang commented on HDFS-8033: - [~drankye] Unfortunately it didn't work. It still tried to apply patches against trunk (the good-looking table was for Walter's small patch, which is compatible with trunk). Maybe it's because I didn't change target version to HDFS-7285 when submitting patch. Will try again later. > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: ByteBufferStrategy_support_len.patch, > HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, > HDFS-8033.003.patch, HDFS-8033.004.patch, HDFS-8033.005.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14510191#comment-14510191 ] Kai Zheng commented on HDFS-8033: - Good work here. Looks like the Jenkins building works now? > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: ByteBufferStrategy_support_len.patch, > HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch, > HDFS-8033.003.patch, hdfs8033-HDFS-7285.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509407#comment-14509407
]
Hadoop QA commented on HDFS-8033:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 42s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | javac | 7m 38s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 49s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 3m 57s | There were no new checkstyle
issues. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 5s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 184m 28s | Tests passed in hadoop-hdfs.
|
| | | 229m 31s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727610/ByteBufferStrategy_support_len.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / baf8bc6 |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10359/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10359/testReport/ |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10359/console |
This message was automatically generated.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: ByteBufferStrategy_support_len.patch,
> HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch,
> HDFS-8033.003.patch, hdfs8033-HDFS-7285.04.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509074#comment-14509074
]
Walter Su commented on HDFS-8033:
-
upload ByteBufferStrategy_support_len.patch. It fixes the issue.
{code}
// Read from one blockReader up to cell boundary
int cellRet = readBuffer(blockReaders[idxInStripe],
currentNodes[idxInStripe], strategy, off + result,
realLenInCell);
{code}
What does The patch do: When bytebuffer.capacity() > realLenInCell, It only
reads realLenInCell.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: ByteBufferStrategy_support_len.patch,
> HDFS-8033.000.patch, HDFS-8033.001.patch, HDFS-8033.002.patch,
> HDFS-8033.003.patch, hdfs8033-HDFS-7285.04.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508937#comment-14508937 ] Walter Su commented on HDFS-8033: - bq. the size is most likely 64K bytes (BlockSender#MIN_BUFFER_WITH_TRANSFERTO. I had an offline discussion with Colin Patrick McCabe about this behavior. Given our default cell size (128K or 256K), if we inherit the behavior from DFSInputStream and return 64K at a time, in most cases we won't cross cell boundary in a single read() anyway. Assume 128K cell size. You give len == 64k, You call read() twice. You get 2 * 64k = 128k. It's ok. Assume MIN_BUFFER_WITH_TRANSFERTO == 96k, and you use bytebuffer. You give len == 64k, You call read() twice. You want 128k, but you get 2 * 96k = 192k. User may not config "io.file.buffer.size" == 4099. But user may config "io.file.buffer.size" == 4k * 3 == 12k, or 4k *5 == 20k. ( Notice that 128k%12k != 0 ) > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch, > HDFS-8033.002.patch, HDFS-8033.003.patch, hdfs8033-HDFS-7285.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508580#comment-14508580
]
Walter Su commented on HDFS-8033:
-
>... ByteBufferStrategy.doRead() ignores len argument. It always read
>byteBuffer.remaining, untils EOF of the current block.
Correct myself:
ByteBufferStrategy.doRead() ignores len argument. It always read
byteBuffer.remaining, untils EOF of the current *packet*.
I read {{BlockSender.doSendBlock()}}. I found out that packet size is depended
by "io.file.buffer.size" and BlockSender.MIN_BUFFER_WITH_TRANSFERTO. If we read
block locally, then size of data part of packet is
"io.file.buffer.size"(default 4096).
HdfsConstants.BLOCK_STRIPED_CELL_SIZE = 256 * 1024;
Good thing is, cellSize%packetSize == 0, 256 * 1024 /4096 == 4; so we call
{{ByteBufferStrategy.doRead()}} 4 times. We can read exactly one cell.
What if cellSize%packetSize != 0? It'll be wrong.
Try config "io.file.buffer.size" == 4099. The testcase will failed. ( any other
value cellSize%packetSize != 0 )
Your implementation for bytebuffer works now. But We have to make sure,
cellSize % ("io.file.buffer.size") ==0 (for local read)
cellSize % (BlockSender.MIN_BUFFER_WITH_TRANSFERTO) ==0 (for remote read)
*When we choose another value for cellSize , we should be careful. Otherwise
read(bytebuffer) won't work.*
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch,
> HDFS-8033.002.patch, HDFS-8033.003.patch, hdfs8033-HDFS-7285.04.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508511#comment-14508511
]
Hadoop QA commented on HDFS-8033:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727544/hdfs8033-HDFS-7285.04.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 18eb5e7 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10357/console |
This message was automatically generated.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch,
> HDFS-8033.002.patch, HDFS-8033.003.patch, hdfs8033-HDFS-7285.04.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508410#comment-14508410
]
Walter Su commented on HDFS-8033:
-
1.
When I run testcase,
{code}
int cellRet = readBuffer(blockReaders[idxInStripe],
currentNodes[idxInStripe], strategy, off + result,
realLenInCell);
{code}
cellRet == 4096 always. It is what you expected? If so, the testcase is not
good enough.
2.
{code}
final int blkIdxInGroup = (int) (startInBlk / cellSize % dataBlkNum);
...
int idxInStripe = (int) ((pos / cellSize) % dataBlkNum);
{code}
How about we call all the the horizonal indexes "idxInGroup"?
3.
{code}
LocatedBlock block = getBlockGroupAt(blockStartOffset);
LocatedStripedBlock blockGroup = getBlockGroupAt(block.getStartOffset());
{code}
are the two lines dulplicated?
4. fatal:
{code}
// Read from one blockReader up to cell boundary
int cellRet = readBuffer(blockReaders[idxInStripe],
currentNodes[idxInStripe], strategy, off + result,
realLenInCell);
{code}
It doesn't work for read(bytebuffer buf).
{{ByteBufferStrategy.doRead()}} ignores {{len}} argument. It always read
byteBuffer.remaining, untils EOF of the current block.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch,
> HDFS-8033.002.patch, HDFS-8033.003.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508196#comment-14508196
]
Zhe Zhang commented on HDFS-8033:
-
Thanks Jing for the review; it's very helpful.
bq. Thus I guess we only need to define a new seekToNextBlock method which have
the same functionality but different signature.
Good point. Besides {{readWithStrategy}}, {{blockSeekTo}} is only used in
{{seekToBlockSource}} and {{seekToNewSource}} and these two are very simple
methods. We should override them anyway.
bq. We can define a different version of readBuffer method which utilizes the
reader list to read. And this new readBuffer method does not need to override
the one in DFSInputStream.
bq. Another reason that we do not reuse the DFSInutStream#readBuffer is that
the retry logic is different. From striped inputstream we do not need to try
another source when hitting any failure.
I agree; a new {{readBuffer}} also makes it easier to implement parallel
stateful read in the future.
[~xinwei] is working on HDFS-8201, which aims to add e2e tests with both
striped input and output streams.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch,
> HDFS-8033.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508177#comment-14508177
]
Jing Zhao commented on HDFS-8033:
-
Thanks for updating the patch, Zhe! Some comments on the 002 patch:
# I think we do not need to override the blockSeekTo method. The return type of
blockSeekTo is a DataNodeInfo, which does not suit striped stream very well.
Thus I guess we only need to define a new {{seekToNextBlock}} method which have
the same functionality but different signature.
# To me {{currentNode}} and {{blockReader}} should be internal states only used
by {{DFSInputStream}}. {{DFSStripedInputStream}} should only use its own
reader/DN list, instead of setting {{currentNode}} and {{blockReader}} and
calling {{DFSInputStream#readBuffer}}. We can define a different version of
{{readBuffer}} method which utilizes the reader list to read. And this new
{{readBuffer}} method does not need to override the one in DFSInputStream.
# Another reason that we do not reuse the {{DFSInutStream#readBuffer}} is that
the retry logic is different. From striped inputstream we do not need to try
another source when hitting any failure.
# We need more tests. Now instead of injecting ec blocks, we can directly use
DFSStripedOutputstream for writing. And then we can compare the reading results
(of both stateful read and pread) with the original written content.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch,
> HDFS-8033.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506131#comment-14506131
]
Yi Liu commented on HDFS-8033:
--
Thanks Zhe for the response, and also thanks Jing for the review and good
comments!
Yes, it's a good idea to maintain a list of current DataNodes.
{quote}
DFSStripedInputStream#readBuffer does switch the blockReader. So after reading
cell_0, we'll switch to the next blockReader and read cell_1.
{quote}
{{DFSStripedInputStream#readBuffer}} can switch the blockReader, that's true,
but it depends on the caller to call, currently for one stateful read,
{{DFSInputStream#readWithStrategy}} only call it one time, so the result only
contains part of the expected data.
I didn't see you do loop (parallelly) all blockReaders for the striped block in
{{DFSStripedInputStream#readBuffer}}.
I have another comments,
I think we need to change {{closeCurrentBlockReader}} to
{{closeCurrentBlockReaders}} in {{DFSInputStream}} and overwrite it in
{{DFSStripedInputStream}}, otherwise there is leak for {{blockReaders}}.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506006#comment-14506006
]
Zhe Zhang commented on HDFS-8033:
-
Continuing with Jing's comments:
bq. In DFSInputStream, the reading state can be captured by blockReader,
currentNode, etc. When reading striped blocks, it may be more clear to define
the state as the group of readers and DataNodes.
Good point about maintaining a list of current DataNodes. This will be helpful
to track bad nodes in the parallel reading case. The patch already maintains
{{blockReaders}}. The entire list of {{blockReader}}s are forwarded to the next
striped block group every time {{blockSeekTo}} is called. I guess in the
parallel stateful reading case we can just read from those {{blockReader}}s in
parallel.
bq. Thus instead of declare blockReader, currentNode, etc as protected and keep
updating them in DFSStripedInputStream, I guess it may be better to define
DFSStripedInputStream's own states (using a list of readers etc.). We may want
to override the whole readWithStrategy method in DFSStripedInputStream as well.
Yes, per the above discussion, we should also override {{readWithStrategy}} to
handle {{currentNodes}}
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505991#comment-14505991
]
Zhe Zhang commented on HDFS-8033:
-
Thanks for the helpful comments Yi and Jing.
bq. In DFSStripedInputStream we override readBuffer, but we only read in one
striped block, so the returned result should be something like (cell_0, cell_3,
) and it only contains part of the expected data
{{DFSStripedInputStream#readBuffer}} does switch the {{blockReader}}. So after
reading cell_0, we'll switch to the next {{blockReader}} and read cell_1.
It's very helpful that you brought up the _short read_ issue. In current
{{DFSInputStream}}, stateful read calls {{blockReader.read()}} once, which
returns all remaining data in the {{blockReader}}'s buffer; the size is most
likely 64K bytes ({{BlockSender#MIN_BUFFER_WITH_TRANSFERTO}}. I had an offline
discussion with [~cmccabe] about this behavior. It seems the rationale is to
return as fast as possible with all cached data. Given our default cell size
(128K or 256K), if we inherit the behavior from {{DFSInputStream}} and return
64K at a time, in most cases we won't cross cell boundary in a single
{{read()}} anyway. So I didn't add the logic of reading across cell boundary in
the patch. It's not too hard to add though, once we make a decision. But I
think we should keep the behavior of trying to return with buffered data
(instead of trying to read up to the request length).
bq. In blockSeekTo, we need to handle refetchToken and refetchEncryptionKey.
And for other IOException, we can throw it.
Good point. Since all EC internal blocks only has 1 destination DN, we won't
have the _while_ loop to count retries. We can retry on different internal
blocks.
bq. For the test, do stateful read: read once and fully read (please make the
data size large than groupSize * cellSize), as I said in #1,
Will test reading multiple {{BLOCK_GROUP_SIZE}} to verify {{blockSeekTo}}
switches between block groups correctly.
bq. connectFailedOnce in blockSeekTo is not necessary.
I agree, will remove it.
bq. Why you modify SimulatedFSDataset?
Once HDFS-8191 is in that won't be needed.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505735#comment-14505735
]
Jing Zhao commented on HDFS-8033:
-
Thanks for working on this, Zhe! And thanks for the review, Yi!
Besides Yi's comments, some other thoughts:
# In {{DFSInputStream}}, the reading state can be captured by {{blockReader}},
{{currentNode}}, etc. When reading striped blocks, it may be more clear to
define the state as the group of readers and DataNodes.
# Besides, the current patch mainly overrides the {{blockSeekTo}} and the
{{readBuffer}} methods. In this way, it may be hard to support parallel reading
or failure handling (both will be handled in separate jiras) in the future.
# Thus instead of declare {{blockReader}}, {{currentNode}}, etc as protected
and keep updating them in {{DFSStripedInputStream}}, I guess it may be better
to define {{DFSStripedInputStream}}'s own states (using a list of readers
etc.). We may want to override the whole {{readWithStrategy}} method in
{{DFSStripedInputStream}} as well.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504485#comment-14504485
]
Yi Liu commented on HDFS-8033:
--
BTW, I find we also need to handle {{seek}}, zerocopy read for
{{DFSStripedInputStream}}, I filed HDFS-8203 to handle it.
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[
https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504428#comment-14504428
]
Yi Liu commented on HDFS-8033:
--
Thanks [~zhz] for working on this. The patch is good, my comments:
*1.*
In DFSInputStream, the stateful read is not to read fully for the output *buf*,
{{readWithStrategy}} will call {{readBuffer}} and return on success. In
{{DFSStripedInputStream}} we override {{readBuffer}}, but we only read in one
striped block, so the returned result should be something like (cell_0, cell_3,
).
This is not incorrect, in the test, you have tested stateful read, but you do
fully read and the data size is *BLOCK_GROUP_SIZE*, so the result
coincidentally is correct.
I suggest we try to do fully read in {{readBuffer}} of
{{DFSStripedInputStream}} unless we find the end of file, of course, the final
read length could be less than the input buf length if we get eof.
*2.* In {{blockSeekTo}}, we need to handle refetchToken and
refetchEncryptionKey. And for other IOException, we can throw it.
*3.*
For the test, do stateful read: read once and fully read (please make the data
size large than groupSize * cellSize), as I said in #1,
*4.*
{{connectFailedOnce}} in {{blockSeekTo}} is not necessary.
*5.*
Why you modify {{SimulatedFSDataset}}?
> Erasure coding: stateful (non-positional) read from files in striped layout
> ---
>
> Key: HDFS-8033
> URL: https://issues.apache.org/jira/browse/HDFS-8033
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14493886#comment-14493886 ] GAO Rui commented on HDFS-8033: --- Zhe, thank you very much for your help. I understand read() and pread() now. > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Zhe Zhang > Attachments: HDFS-8033.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14391026#comment-14391026 ] Zhe Zhang commented on HDFS-8033: - Thanks for the question Rui. Similar to UNIX, HDFS provides both _read_ and _pread_ semantics. In a nutshell, _pread_ doesn't modify the state of the file descriptor, like the current position. For example, when you do _read()_, _pread()_, _read()_, then second _read()_ will resume from the position after the first _read()_. Please also refer to [this | http://stackoverflow.com/questions/1687275/what-is-the-difference-between-read-and-pread-in-unix] > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Zhe Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
[ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389891#comment-14389891 ] GAO Rui commented on HDFS-8033: --- [~zhz] bq. stateful (non-positional) read means reading the whole file without any position requirement? > Erasure coding: stateful (non-positional) read from files in striped layout > --- > > Key: HDFS-8033 > URL: https://issues.apache.org/jira/browse/HDFS-8033 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Zhe Zhang >Assignee: Zhe Zhang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
