[jira] [Commented] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

2015-09-21 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901197#comment-14901197 ] Jing Zhao commented on HDFS-9040: - bq. In short, bumpGS is useful for choosing working set(

[jira] [Assigned] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

2015-09-21 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao reassigned HDFS-9040: --- Assignee: Jing Zhao > Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests > t

[jira] [Commented] (HDFS-9106) Transfer failure during pipeline recovery causes permanent write failures

2015-09-21 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901442#comment-14901442 ] Jing Zhao commented on HDFS-9106: - bq. Transfer timeout needs to be different from per-pack

[jira] [Commented] (HDFS-9112) Haadmin fails if multiple name service IDs are configured

2015-09-21 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901549#comment-14901549 ] Jing Zhao commented on HDFS-9112: - We had a discussion in HDFS-6376 about this and [~dlmari

[jira] [Commented] (HDFS-9112) Haadmin fails if multiple name service IDs are configured

2015-09-21 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901613#comment-14901613 ] Jing Zhao commented on HDFS-9112: - Thanks for the clarification, [~anu]! I think for admin

[jira] [Commented] (HDFS-9053) Support large directories efficiently using B-Tree

2015-09-21 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901761#comment-14901761 ] Jing Zhao commented on HDFS-9053: - Thanks for the great work, Yi! So far I just reviewed th

[jira] [Updated] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

2015-09-22 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9040: Attachment: HDFS-9040-HDFS-7285.004.patch Update the patch based on our discussion. The main changes compared

[jira] [Commented] (HDFS-9112) Haadmin fails if multiple name service IDs are configured

2015-09-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905137#comment-14905137 ] Jing Zhao commented on HDFS-9112: - Thanks for the new patch, [~anu]! And thanks for the rev

[jira] [Commented] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

2015-09-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905302#comment-14905302 ] Jing Zhao commented on HDFS-9040: - Thanks for the review, [~walter.k.su]! bq. It's not goo

[jira] [Updated] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

2015-09-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9040: Attachment: HDFS-9040-HDFS-7285.005.patch Update the patch to address Walter's comments. > Erasure coding: R

[jira] [Commented] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly

2015-09-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905506#comment-14905506 ] Jing Zhao commented on HDFS-1172: - Any progress [~iwasakims] and [~walter.k.su]? > Blocks

[jira] [Updated] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

2015-09-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9040: Attachment: HDFS-9040-HDFS-7285.005.patch Rebase the patch against the latest feature branch. > Erasure codi

[jira] [Commented] (HDFS-9079) Erasure coding: preallocate multiple generation stamps and serialize updates from data streamers

2015-09-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905562#comment-14905562 ] Jing Zhao commented on HDFS-9079: - Thanks for working on this, [~zhz]! To have an event qu

[jira] [Updated] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

2015-09-23 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9040: Attachment: HDFS-9040-HDFS-7285.006.patch bq. How about wait at most halfSocketTimeout? Good idea! Update th

[jira] [Commented] (HDFS-9112) Haadmin fails if multiple name service IDs are configured

2015-09-24 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906946#comment-14906946 ] Jing Zhao commented on HDFS-9112: - Thanks for updating the patch, [~anu]! The latest patch

[jira] [Created] (HDFS-9145) Tracking methods that hold FSNamesytemLock for too long

2015-09-25 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-9145: --- Summary: Tracking methods that hold FSNamesytemLock for too long Key: HDFS-9145 URL: https://issues.apache.org/jira/browse/HDFS-9145 Project: Hadoop HDFS Issue Type: I

[jira] [Comment Edited] (HDFS-7663) Erasure Coding: lease recovery / append on striped file

2015-09-25 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908505#comment-14908505 ] Jing Zhao edited comment on HDFS-7663 at 9/25/15 6:53 PM: -- Thanks

[jira] [Commented] (HDFS-7663) Erasure Coding: lease recovery / append on striped file

2015-09-25 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908505#comment-14908505 ] Jing Zhao commented on HDFS-7663: - Thanks for sharing the thoughts, Walter! First I complet

[jira] [Comment Edited] (HDFS-7663) Erasure Coding: lease recovery / append on striped file

2015-09-25 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908505#comment-14908505 ] Jing Zhao edited comment on HDFS-7663 at 9/25/15 6:54 PM: -- Thanks

[jira] [Updated] (HDFS-9112) Improve error message for Haadmin when multiple name service IDs are configured

2015-09-25 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9112: Issue Type: Improvement (was: Bug) > Improve error message for Haadmin when multiple name service IDs are >

[jira] [Updated] (HDFS-9112) Improve error message for Haadmin when multiple name service IDs are configured

2015-09-25 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9112: Summary: Improve error message for Haadmin when multiple name service IDs are configured (was: Haadmin fails

[jira] [Updated] (HDFS-9112) Improve error message for Haadmin when multiple name service IDs are configured

2015-09-25 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9112: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: P

[jira] [Updated] (HDFS-9145) Tracking methods that hold FSNamesytemLock for too long

2015-09-25 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9145: Description: It will be helpful that if we can have a way to track (or at least log a msg) if some operation

[jira] [Commented] (HDFS-9053) Support large directories efficiently using B-Tree

2015-09-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933696#comment-14933696 ] Jing Zhao commented on HDFS-9053: - Yeah, sorry for the delay. I'm currently reviewing the r

[jira] [Commented] (HDFS-9106) Transfer failure during pipeline recovery causes permanent write failures

2015-09-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933698#comment-14933698 ] Jing Zhao commented on HDFS-9106: - +1 for the latest patch. Thanks [~kihwal]. > Transfer f

[jira] [Commented] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

2015-09-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933703#comment-14933703 ] Jing Zhao commented on HDFS-9040: - [~walter.k.su] and [~zhz], any further comments on the p

[jira] [Commented] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly

2015-09-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933772#comment-14933772 ] Jing Zhao commented on HDFS-1172: - bq. BlockManager#hasEnoughEffectiveReplicas added by HDF

[jira] [Commented] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

2015-09-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934036#comment-14934036 ] Jing Zhao commented on HDFS-9040: - Thanks for the review, Zhe! For {{getFollowingBlock}}, c

[jira] [Updated] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

2015-09-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9040: Attachment: HDFS-9040-HDFS-7285.007.patch Update the patch to address Zhe's comments. > Erasure coding: Refa

[jira] [Updated] (HDFS-9040) Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests to Coordinator)

2015-09-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9040: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7285 Status: Resolved (wa

[jira] [Commented] (HDFS-7663) Erasure Coding: lease recovery / append on striped file

2015-09-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934206#comment-14934206 ] Jing Zhao commented on HDFS-7663: - bq. They have strong relationship when appending is to a

[jira] [Issue Comment Deleted] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks

2015-09-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9070: Comment: was deleted (was: \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime

[jira] [Commented] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks

2015-09-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934303#comment-14934303 ] Jing Zhao commented on HDFS-9070: - Thanks for updating the patch, [~demongaorui]! The new p

[jira] [Commented] (HDFS-9053) Support large directories efficiently using B-Tree

2015-09-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934412#comment-14934412 ] Jing Zhao commented on HDFS-9053: - Some further comments: # In {{INodeDirectory#replaceChil

[jira] [Commented] (HDFS-9053) Support large directories efficiently using B-Tree

2015-09-28 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934414#comment-14934414 ] Jing Zhao commented on HDFS-9053: - Also, since this is a critical part of the code, maybe w

[jira] [Updated] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly

2015-09-29 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-1172: Assignee: Masatake Iwasaki > Blocks in newly completed files are considered under-replicated too quickly > --

[jira] [Commented] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly

2015-09-29 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935973#comment-14935973 ] Jing Zhao commented on HDFS-1172: - # It is not necessary to call {{numNodes}} again. We can

[jira] [Comment Edited] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly

2015-09-29 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935973#comment-14935973 ] Jing Zhao edited comment on HDFS-1172 at 9/29/15 9:56 PM: -- Thanks

[jira] [Created] (HDFS-9180) Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers

2015-09-30 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-9180: --- Summary: Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers Key: HDFS-9180 URL: https://issues.apache.org/jira/browse/HDFS-9180 Project: Ha

[jira] [Updated] (HDFS-9180) Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers

2015-09-30 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9180: Status: Patch Available (was: Open) > Update excluded DataNodes in DFSStripedOutputStream based on failures

[jira] [Updated] (HDFS-9180) Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers

2015-09-30 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9180: Attachment: HDFS-9180.000.patch Upload the first patch. The patch also fixes several bugs: # When replacing a

[jira] [Commented] (HDFS-9185) TestRecoverStripedFile is failing

2015-10-01 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940200#comment-14940200 ] Jing Zhao commented on HDFS-9185: - Thanks for working on this, [~rakeshr]. The changes look

[jira] [Updated] (HDFS-9180) Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers

2015-10-01 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9180: Attachment: HDFS-9180.001.patch Update the patch: # The above fix #3 is unnecessary since we mark a data stre

[jira] [Commented] (HDFS-9180) Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers

2015-10-01 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940564#comment-14940564 ] Jing Zhao commented on HDFS-9180: - To help testing the patch, we can add the following test

[jira] [Commented] (HDFS-9185) TestRecoverStripedFile is failing

2015-10-02 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941492#comment-14941492 ] Jing Zhao commented on HDFS-9185: - The new patch looks good to me. All the failed tests pas

[jira] [Updated] (HDFS-9185) Fix null tracer in ErasureCodingWorker

2015-10-02 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9185: Summary: Fix null tracer in ErasureCodingWorker (was: TestRecoverStripedFile is failing) > Fix null tracer

[jira] [Updated] (HDFS-9185) Fix null tracer in ErasureCodingWorker

2015-10-02 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9185: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: P

[jira] [Updated] (HDFS-9180) Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers

2015-10-02 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9180: Attachment: HDFS-9180.002.patch Thanks for the review, Yi! The failed EC related tests are mainly caused by

[jira] [Commented] (HDFS-9180) Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers

2015-10-02 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941986#comment-14941986 ] Jing Zhao commented on HDFS-9180: - {{TestDFSStripedOutputStreamWithFailure#testDatanodeFail

[jira] [Updated] (HDFS-9180) Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers

2015-10-05 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9180: Attachment: HDFS-9180.003.patch Thanks for the review, Walter and Yi! Update the patch to address Walter's c

[jira] [Commented] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly

2015-10-05 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944105#comment-14944105 ] Jing Zhao commented on HDFS-1172: - bq. I'm trying to update the pendingReplications only in

[jira] [Commented] (HDFS-8966) Separate the lock used in namespace and block management layer

2015-10-05 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944224#comment-14944224 ] Jing Zhao commented on HDFS-8966: - Based on the discussion in HDFS-8967, it makes more sens

[jira] [Updated] (HDFS-8966) Separate the lock used in namespace and block management layer

2015-10-05 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8966: Target Version/s: HDFS-8966 > Separate the lock used in namespace and block management layer > --

[jira] [Commented] (HDFS-9180) Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers

2015-10-05 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944529#comment-14944529 ] Jing Zhao commented on HDFS-9180: - Thanks Walter! [~hitliuyi], could you please take a lo

[jira] [Updated] (HDFS-9180) Update excluded DataNodes in DFSStripedOutputStream based on failures in data streamers

2015-10-06 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9180: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: P

[jira] [Commented] (HDFS-9182) Cleanup the findbugs and other issues after HDFS EC merged to trunk.

2015-10-06 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945478#comment-14945478 ] Jing Zhao commented on HDFS-9182: - Thanks for all the fix, [~umamaheswararao]! The patch ma

[jira] [Commented] (HDFS-9129) Move the safemode block count into BlockManager

2015-10-06 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945578#comment-14945578 ] Jing Zhao commented on HDFS-9129: - The patch looks good overall. Some early comments: # The

[jira] [Created] (HDFS-9204) DatanodeDescriptor#PendingReplicationWithoutTargets is wrongly calculated

2015-10-06 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-9204: --- Summary: DatanodeDescriptor#PendingReplicationWithoutTargets is wrongly calculated Key: HDFS-9204 URL: https://issues.apache.org/jira/browse/HDFS-9204 Project: Hadoop HDFS

[jira] [Updated] (HDFS-9204) DatanodeDescriptor#PendingReplicationWithoutTargets is wrongly calculated

2015-10-06 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9204: Description: This seems to be a regression caused by the merge of EC feature branch. {{DatanodeDescriptor#in

[jira] [Updated] (HDFS-8967) Create a BlockManagerLock class to represent the lock used in the BlockManager

2015-10-06 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8967: Attachment: HDFS-8967.003.patch Rebase the patch. > Create a BlockManagerLock class to represent the lock us

[jira] [Commented] (HDFS-9206) Inconsistent default value of dfs.datanode.stripedread.buffer.size

2015-10-06 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946208#comment-14946208 ] Jing Zhao commented on HDFS-9206: - +1. I will commit this after Jenkins approves. > Incons

[jira] [Updated] (HDFS-8967) Create a BlockManagerLock class to represent the lock used in the BlockManager

2015-10-06 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8967: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-8966 Status: Resolved (wa

[jira] [Updated] (HDFS-9206) Inconsistent default value of dfs.datanode.stripedread.buffer.size

2015-10-06 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9206: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: P

[jira] [Updated] (HDFS-9206) Inconsistent default value of dfs.datanode.stripedread.buffer.size

2015-10-06 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9206: Priority: Minor (was: Trivial) > Inconsistent default value of dfs.datanode.stripedread.buffer.size > --

[jira] [Updated] (HDFS-9196) Fix TestWebHdfsContentLength

2015-10-06 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9196: Summary: Fix TestWebHdfsContentLength (was: TestWebHdfsContentLength fails on trunk) > Fix TestWebHdfsConte

[jira] [Updated] (HDFS-9196) TestWebHdfsContentLength fails on trunk

2015-10-06 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9196: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: P

[jira] [Commented] (HDFS-9170) Move libhdfs / fuse-dfs / libwebhdfs to hdfs-client

2015-10-07 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14946380#comment-14946380 ] Jing Zhao commented on HDFS-9170: - The test failure should be unrelated. +1. > Move libhdf

[jira] [Commented] (HDFS-9209) Erasure coding: Add apache license header in TestFileStatusWithECPolicy.java

2015-10-07 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14947255#comment-14947255 ] Jing Zhao commented on HDFS-9209: - +1. I will commit the patch after the Jenkins run. > Er

[jira] [Commented] (HDFS-9204) DatanodeDescriptor#PendingReplicationWithoutTargets is wrongly calculated

2015-10-07 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14947531#comment-14947531 ] Jing Zhao commented on HDFS-9204: - The patch looks good to me. Some minors: # We can add an

[jira] [Commented] (HDFS-9205) Do not sehedule corrupted blocks for replication

2015-10-07 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14947566#comment-14947566 ] Jing Zhao commented on HDFS-9205: - I think it makes sense to skip scheduling replication fo

[jira] [Updated] (HDFS-9209) Erasure coding: Add apache license header in TestFileStatusWithECPolicy.java

2015-10-07 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9209: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: P

[jira] [Commented] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly

2015-10-08 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949123#comment-14949123 ] Jing Zhao commented on HDFS-1172: - Thanks for updating the patch, [~iwasakims]. The 012 pat

[jira] [Commented] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks

2015-10-08 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949302#comment-14949302 ] Jing Zhao commented on HDFS-9070: - Thanks for updating the patch, [~demongaorui]. The new p

[jira] [Updated] (HDFS-9204) DatanodeDescriptor#PendingReplicationWithoutTargets is wrongly calculated

2015-10-08 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9204: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: P

[jira] [Commented] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks

2015-10-09 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14950834#comment-14950834 ] Jing Zhao commented on HDFS-9070: - bq. We can extract the existing finalized-block's code f

[jira] [Commented] (HDFS-9145) Tracking methods that hold FSNamesytemLock for too long

2015-10-09 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14950894#comment-14950894 ] Jing Zhao commented on HDFS-9145: - I think here we do not need to use a stack to track mult

[jira] [Commented] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly

2015-10-09 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14950942#comment-14950942 ] Jing Zhao commented on HDFS-1172: - Thanks [~iwasakims]! The 013 patch looks pretty good to

[jira] [Commented] (HDFS-9173) Erasure Coding: Lease recovery for striped file

2015-10-09 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14951004#comment-14951004 ] Jing Zhao commented on HDFS-9173: - Thanks a lot for working on this, [~walter.k.su]! As [~z

[jira] [Created] (HDFS-9223) Code cleanup for DatanodeDescriptor and HeartbeatManager

2015-10-09 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-9223: --- Summary: Code cleanup for DatanodeDescriptor and HeartbeatManager Key: HDFS-9223 URL: https://issues.apache.org/jira/browse/HDFS-9223 Project: Hadoop HDFS Issue Type:

[jira] [Updated] (HDFS-9223) Code cleanup for DatanodeDescriptor and HeartbeatManager

2015-10-09 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9223: Attachment: HDFS-9223.000.patch The patch also makes the following changes in {{heartbeatCheck}}. {code} -

[jira] [Updated] (HDFS-9223) Code cleanup for DatanodeDescriptor and HeartbeatManager

2015-10-09 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9223: Status: Patch Available (was: Open) > Code cleanup for DatanodeDescriptor and HeartbeatManager > ---

[jira] [Updated] (HDFS-9223) Code cleanup for DatanodeDescriptor and HeartbeatManager

2015-10-12 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9223: Attachment: HDFS-9223.001.patch Thanks for the review, Nicholas! Update patch to address your comments. Also

[jira] [Commented] (HDFS-9223) Code cleanup for DatanodeDescriptor and HeartbeatManager

2015-10-12 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953715#comment-14953715 ] Jing Zhao commented on HDFS-9223: - For {{heartbeatCheck}}, currently the {{dead}} node obje

[jira] [Commented] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly

2015-10-12 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953717#comment-14953717 ] Jing Zhao commented on HDFS-1172: - +1 for the 014 patch. I will commit it later today or ea

[jira] [Commented] (HDFS-9145) Tracking methods that hold FSNamesytemLock for too long

2015-10-12 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953755#comment-14953755 ] Jing Zhao commented on HDFS-9145: - The patch looks good to me. One minor optional thing to

[jira] [Commented] (HDFS-9205) Do not schedule corrupt blocks for replication

2015-10-12 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954147#comment-14954147 ] Jing Zhao commented on HDFS-9205: - # Nit: need to fix the javadoc of {{UnderReplicatedBlock

[jira] [Commented] (HDFS-8287) DFSStripedOutputStream.writeChunk should not wait for writing parity

2015-10-12 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954156#comment-14954156 ] Jing Zhao commented on HDFS-8287: - I'm very sorry I missed this jira in my radar.. Could yo

[jira] [Commented] (HDFS-8438) Erasure Coding: Allow concat striped files if they have the same ErasureCodingPolicy

2015-10-13 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955361#comment-14955361 ] Jing Zhao commented on HDFS-8438: - +1. I will commit the patch shortly. > Erasure Coding:

[jira] [Updated] (HDFS-8438) Erasure Coding: Allow concat striped files if they have the same ErasureCodingPolicy

2015-10-13 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-8438: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: P

[jira] [Updated] (HDFS-9223) Code cleanup for DatanodeDescriptor and HeartbeatManager

2015-10-13 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9223: Attachment: HDFS-9223.002.patch Update the patch to address Nicholas's comments. > Code cleanup for Datanode

[jira] [Updated] (HDFS-9220) Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum

2015-10-13 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9220: Priority: Critical (was: Major) > Reading small file (< 512 bytes) that is open for append fails due to > i

[jira] [Commented] (HDFS-9220) Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum

2015-10-13 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955447#comment-14955447 ] Jing Zhao commented on HDFS-9220: - Looks like the following change can fix the issue: {code

[jira] [Commented] (HDFS-9145) Tracking methods that hold FSNamesytemLock for too long

2015-10-13 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955560#comment-14955560 ] Jing Zhao commented on HDFS-9145: - +1 > Tracking methods that hold FSNamesytemLock for too

[jira] [Commented] (HDFS-8779) WebUI can't display randomly generated block ID

2015-10-13 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955642#comment-14955642 ] Jing Zhao commented on HDFS-8779: - +1 on the 006 patch > WebUI can't display randomly gene

[jira] [Commented] (HDFS-9173) Erasure Coding: Lease recovery for striped file

2015-10-13 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955936#comment-14955936 ] Jing Zhao commented on HDFS-9173: - The patch looks pretty good to me. Some comments: # The

[jira] [Updated] (HDFS-9220) Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum

2015-10-13 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9220: Attachment: HDFS-9220.000.patch Submit a patch including the above fix and also [~bograd]'s unit test. > Rea

[jira] [Updated] (HDFS-9220) Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum

2015-10-13 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9220: Status: Patch Available (was: Open) > Reading small file (< 512 bytes) that is open for append fails due to

[jira] [Updated] (HDFS-9220) Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum

2015-10-13 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9220: Target Version/s: 2.7.2 > Reading small file (< 512 bytes) that is open for append fails due to > incorrect

[jira] [Updated] (HDFS-9220) Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum

2015-10-13 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9220: Priority: Blocker (was: Critical) > Reading small file (< 512 bytes) that is open for append fails due to >

[jira] [Updated] (HDFS-9220) Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum

2015-10-13 Thread Jing Zhao (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-9220: Attachment: HDFS-9220.001.patch Update the patch to address [~templedf]'s comments. > Reading small file (<

<    1   2   3   4   5   6   7   8   9   10   >