[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938014#comment-14938014 ] Hudson commented on HDFS-7889: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937795#comment-14937795 ] Hudson commented on HDFS-7889: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938390#comment-14938390 ] Hudson commented on HDFS-7889: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938584#comment-14938584 ] Hudson commented on HDFS-7889: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937538#comment-14937538 ] Hudson commented on HDFS-7889: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937288#comment-14937288 ] Hudson commented on HDFS-7889: -- FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937093#comment-14937093 ] Hudson commented on HDFS-7889: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499256#comment-14499256 ] Zhe Zhang commented on HDFS-7889: - OK I think I get the logic now. However, it is fragile

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494495#comment-14494495 ] Zhe Zhang commented on HDFS-7889: - As we go deeper in testing, I've found the following

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495614#comment-14495614 ] Li Bo commented on HDFS-7889: - hi, Zhe, please see my following explanation of the related

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490528#comment-14490528 ] Zhe Zhang commented on HDFS-7889: - Thanks Bo for completing this challenging task!

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14487041#comment-14487041 ] Li Bo commented on HDFS-7889: - Changes in patch 010: check file length in unit test; check if

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482890#comment-14482890 ] Li Bo commented on HDFS-7889: - hi, Zhe Thanks for your review and update of patch. 1.

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484383#comment-14484383 ] Zhe Zhang commented on HDFS-7889: - bq. in encode(), just copy the references of buffer in

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482606#comment-14482606 ] Zhe Zhang commented on HDFS-7889: - Would also be nice to verify the parity data content in

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394178#comment-14394178 ] Li Bo commented on HDFS-7889: - Patch 008 fix problems mentioned above. Currently I have not

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394148#comment-14394148 ] Walter Su commented on HDFS-7889: - HDFS-7839 removed HdfsConstants.EC_STORAGE_POLICY_ID

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394134#comment-14394134 ] Li Bo commented on HDFS-7889: - OK, I will upload a new patch later Subclass DFSOutputStream

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392260#comment-14392260 ] Vinayakumar B commented on HDFS-7889: - {quote}Moving newStreamForCreate to DFSClient: I

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392394#comment-14392394 ] Li Bo commented on HDFS-7889: - In patch 007, we will not pad zero-bytes to incomplete stripe

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392245#comment-14392245 ] Vinayakumar B commented on HDFS-7889: - Hi [~libo-intel], thanks for the nice work.

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392385#comment-14392385 ] Li Bo commented on HDFS-7889: - Thanks for Vinayakumar’s careful review. bq. I didn't understand

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394095#comment-14394095 ] Zhe Zhang commented on HDFS-7889: - {{writeChunkImpl}} was removed from {{DFSOutputStream}}

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392493#comment-14392493 ] Vinayakumar B commented on HDFS-7889: - bq. This is because

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14393051#comment-14393051 ] Zhe Zhang commented on HDFS-7889: - Thanks Bo for the update and Vinay for the thorough

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388090#comment-14388090 ] Li Bo commented on HDFS-7889: - hi, Zhe {{stripedBlocks[i]}} is an instance of

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389987#comment-14389987 ] Li Bo commented on HDFS-7889: - Patch 006 removes {{getStreamer()}} and switches streamer in

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386476#comment-14386476 ] Li Bo commented on HDFS-7889: - Thanks for Zhe’s review and update of the patch. bq. we can

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387204#comment-14387204 ] Zhe Zhang commented on HDFS-7889: - Thanks Bo for the revised patch! I think the main

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387999#comment-14387999 ] Li Bo commented on HDFS-7889: - Thanks for Zhe's review. Changes irrelevant to subclassing have

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388064#comment-14388064 ] Zhe Zhang commented on HDFS-7889: - Thanks Bo! bq. If not padding zero-bytes to incomplete

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381686#comment-14381686 ] Hadoop QA commented on HDFS-7889: - {color:red}-1 overall{color}. Here are the results of

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1438#comment-1438 ] Hadoop QA commented on HDFS-7889: - {color:red}-1 overall{color}. Here are the results of

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383282#comment-14383282 ] Li Bo commented on HDFS-7889: - Thanks Zhe’s careful review. bq. Isn’t currentPacket always null

[jira] [Commented] (HDFS-7889) Subclass DFSOutputStream to support writing striping layout files

[ https://issues.apache.org/jira/browse/HDFS-7889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14382364#comment-14382364 ] Zhe Zhang commented on HDFS-7889: - Thanks Bo for the patch! The main striped writing logic