[jira] [Updated] (HDFS-11817) A faulty node can cause a lease leak and NPE on accessing data

2017-05-18 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11817: -- Attachment: hdfs-11817_supplement.txt Attaching supplemental information including stack traces. > A fa

[jira] [Commented] (HDFS-11817) A faulty node can cause a lease leak and NPE on accessing data

2017-05-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014712#comment-16014712 ] Kihwal Lee commented on HDFS-11817: --- *Summary:* The observation of the incident resulte

[jira] [Commented] (HDFS-11817) A faulty node can cause a lease leak and NPE on accessing data

2017-05-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014703#comment-16014703 ] Kihwal Lee commented on HDFS-11817: --- *Details of the NPE:* The JVM did produce a stacktr

[jira] [Updated] (HDFS-11641) Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus

2017-05-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11641: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.2 3.0.0-alpha3

[jira] [Commented] (HDFS-11641) Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus

2017-05-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16012686#comment-16012686 ] Kihwal Lee commented on HDFS-11641: --- Please provide a patch for branch-2. With the EC-r

[jira] [Commented] (HDFS-11641) Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus

2017-05-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16012671#comment-16012671 ] Kihwal Lee commented on HDFS-11641: --- +1 the latest patch looks good. > Reduce cost of a

[jira] [Updated] (HDFS-11641) Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus

2017-05-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11641: -- Target Version/s: 2.8.2 (was: 2.8.1) > Reduce cost of audit logging by using FileStatus instead of Hdfs

[jira] [Commented] (HDFS-11820) Thread safety in logEdit?

2017-05-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011498#comment-16011498 ] Kihwal Lee commented on HDFS-11820: --- bq. Thread #2 catches up, and acquires same instanc

[jira] [Commented] (HDFS-11820) Thread safety in logEdit?

2017-05-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011457#comment-16011457 ] Kihwal Lee commented on HDFS-11820: --- Write lock ensures there is only one thread calling

[jira] [Resolved] (HDFS-11820) Thread safety in logEdit?

2017-05-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-11820. --- Resolution: Invalid > Thread safety in logEdit? > - > > Key: H

[jira] [Commented] (HDFS-11820) Thread safety in logEdit?

2017-05-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010962#comment-16010962 ] Kihwal Lee commented on HDFS-11820: --- Jira is for bug report. Please use relevant mailing

[jira] [Commented] (HDFS-11817) A faulty node can cause a lease leak and NPE on accessing data

2017-05-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010646#comment-16010646 ] Kihwal Lee commented on HDFS-11817: --- There are two possible immediate fixes that can be

[jira] [Assigned] (HDFS-11817) A faulty node can cause a lease leak and NPE on accessing data

2017-05-12 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-11817: - Assignee: Kihwal Lee > A faulty node can cause a lease leak and NPE on accessing data > -

[jira] [Created] (HDFS-11817) A faulty node can cause a lease leak and NPE on accessing data

2017-05-12 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11817: - Summary: A faulty node can cause a lease leak and NPE on accessing data Key: HDFS-11817 URL: https://issues.apache.org/jira/browse/HDFS-11817 Project: Hadoop HDFS

[jira] [Updated] (HDFS-11755) Underconstruction blocks can be considered missing

2017-05-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11755: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.2 3.0.0-alpha3

[jira] [Commented] (HDFS-11755) Underconstruction blocks can be considered missing

2017-05-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005219#comment-16005219 ] Kihwal Lee commented on HDFS-11755: --- +1 looks good. > Underconstruction blocks can be c

[jira] [Commented] (HDFS-8789) Block Placement policy migrator

2017-05-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004804#comment-16004804 ] Kihwal Lee commented on HDFS-8789: -- What happens on NN restart (to be precise, during the

[jira] [Updated] (HDFS-11765) Fix:Performance regression due to incorrect use of DataChecksum

2017-05-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11765: -- Target Version/s: 2.8.1 > Fix:Performance regression due to incorrect use of DataChecksum >

[jira] [Commented] (HDFS-8789) Block Placement policy migrator

2017-05-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001021#comment-16001021 ] Kihwal Lee commented on HDFS-8789: -- This can be fatal to namenode. {code} blocksWlocs = n

[jira] [Updated] (HDFS-11702) Remove indefinite caching of key provider uri in DFSClient

2017-05-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11702: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.2 3.0.0-alpha3

[jira] [Commented] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999105#comment-15999105 ] Kihwal Lee commented on HDFS-11742: --- bq. I probably still not. I invite everyone to run

[jira] [Comment Edited] (HDFS-11742) Improve balancer usability after HDFS-8818

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999071#comment-15999071 ] Kihwal Lee edited comment on HDFS-11742 at 5/5/17 11:24 PM: bq

[jira] [Updated] (HDFS-11742) Improve balancer usability after HDFS-8818

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Summary: Improve balancer usability after HDFS-8818 (was: Improve balancer usability after HDFS-8188)

[jira] [Commented] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999071#comment-15999071 ] Kihwal Lee commented on HDFS-11742: --- bq. Do you think that this is still a blocker of t

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998564#comment-15998564 ] Kihwal Lee commented on HDFS-8818: -- Posted a patch to HDFS-11742. Hopefully, this will pr

[jira] [Updated] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Attachment: HDFS-11742.v2.trunk.patch > Improve balancer usability after HDFS-8188 > ---

[jira] [Commented] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998536#comment-15998536 ] Kihwal Lee commented on HDFS-11742: --- Setting HDFS-11377 as a dependency. In most cases,

[jira] [Commented] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998534#comment-15998534 ] Kihwal Lee commented on HDFS-11742: --- Instead of reverting, I am making a simple change t

[jira] [Updated] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Description: We ran 2.8 balancer with HDFS-8818 on a 280-node and a 2,400-node cluster. In both cases,

[jira] [Updated] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Target Version/s: 2.7.4, 2.8.1 (was: 2.8.1) > Improve balancer usability after HDFS-8188 >

[jira] [Updated] (HDFS-11742) Improve balancer usability after HDFS-8188

2017-05-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Summary: Improve balancer usability after HDFS-8188 (was: Revert the core changes from HDFS-8818) > Im

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-04 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997774#comment-15997774 ] Kihwal Lee commented on HDFS-8818: -- I will post a patch for improvement tomorrow. I am let

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-04 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996738#comment-15996738 ] Kihwal Lee commented on HDFS-8818: -- bq. Do you have HDFS-8824 in your runs? I suspect th

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995477#comment-15995477 ] Kihwal Lee commented on HDFS-8818: -- I set the dfs.datanode.balance.max.concurrent.moves to

[jira] [Comment Edited] (HDFS-8818) Allow Balancer to run faster

2017-05-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995477#comment-15995477 ] Kihwal Lee edited comment on HDFS-8818 at 5/3/17 7:17 PM: -- I set d

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994840#comment-15994840 ] Kihwal Lee commented on HDFS-8818: -- We see this problem in EVERY single cluster we tried t

[jira] [Updated] (HDFS-8818) Allow Balancer to run faster

2017-05-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: bal1.png bal2.png > Allow Balancer to run faster >

[jira] [Comment Edited] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994001#comment-15994001 ] Kihwal Lee edited comment on HDFS-11742 at 5/2/17 11:26 PM: !h

[jira] [Commented] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994001#comment-15994001 ] Kihwal Lee commented on HDFS-11742: --- !https://issues.apache.org/jira/secure/attachment/1

[jira] [Updated] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Attachment: balancer2.8.png > Revert the core changes from HDFS-8818 > -

[jira] [Updated] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Description: This is to revert the core changes made by HDFS-8818. The reason is explained in the jira

[jira] [Updated] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Assignee: Kihwal Lee Status: Patch Available (was: Open) > Revert the core changes from HDFS-8818

[jira] [Updated] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11742: -- Attachment: HDFS-11742.trunk.patch HDFS-11742.branch-2.patch HDFS-11742.b

[jira] [Updated] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: (was: bal_revert.branch-2.patch) > Allow Balancer to run faster > -

[jira] [Updated] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: (was: bal_revert.trunk.patch) > Allow Balancer to run faster >

[jira] [Updated] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: (was: bal_revert.branch-2.8.patch) > Allow Balancer to run faster > ---

[jira] [Created] (HDFS-11742) Revert the core changes from HDFS-8818

2017-05-02 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11742: - Summary: Revert the core changes from HDFS-8818 Key: HDFS-11742 URL: https://issues.apache.org/jira/browse/HDFS-11742 Project: Hadoop HDFS Issue Type: Bug

[jira] [Resolved] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-8818. -- Resolution: Fixed > Allow Balancer to run faster > > > Key:

[jira] [Updated] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8818: - Attachment: bal_revert.trunk.patch bal_revert.branch-2.patch bal_revert.bran

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993840#comment-15993840 ] Kihwal Lee commented on HDFS-8818: -- I will open a new one and post the patch there. > All

[jira] [Reopened] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reopened HDFS-8818: -- Reopening for revert. git won't revert it cleanly since there have been other changes since the commit. I m

[jira] [Assigned] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-8818: Assignee: Kihwal Lee (was: Tsz Wo Nicholas Sze) > Allow Balancer to run faster > --

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993723#comment-15993723 ] Kihwal Lee commented on HDFS-8818: -- Thanks for the update, [~jojochuang]. That fixes the h

[jira] [Commented] (HDFS-8818) Allow Balancer to run faster

2017-05-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993654#comment-15993654 ] Kihwal Lee commented on HDFS-8818: -- -1. Proposing revert of this untested change. We've r

[jira] [Updated] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11714: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.1 3.0.0-alpha3

[jira] [Comment Edited] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991670#comment-15991670 ] Kihwal Lee edited comment on HDFS-11714 at 5/1/17 10:24 PM: th

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991670#comment-15991670 ] Kihwal Lee commented on HDFS-11714: --- the branch-2 patch is identical except for the vari

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991665#comment-15991665 ] Kihwal Lee commented on HDFS-11714: --- You beat me. Thanks. > Newly added NN storage dire

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991664#comment-15991664 ] Kihwal Lee commented on HDFS-11714: --- TestPipelinesFailover, TestBalancer are passing. Th

[jira] [Commented] (HDFS-7541) Upgrade Domains in HDFS

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991533#comment-15991533 ] Kihwal Lee commented on HDFS-7541: -- Just found out I somehow clicked the evil "Assign to m

[jira] [Assigned] (HDFS-7541) Upgrade Domains in HDFS

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-7541: Assignee: Ming Ma (was: Kihwal Lee) > Upgrade Domains in HDFS > --- > >

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991469#comment-15991469 ] Kihwal Lee commented on HDFS-11714: --- Filed HDFS-11729. > Newly added NN storage directo

[jira] [Created] (HDFS-11729) Improve NNStorageRetentionManager failure handling.

2017-05-01 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11729: - Summary: Improve NNStorageRetentionManager failure handling. Key: HDFS-11729 URL: https://issues.apache.org/jira/browse/HDFS-11729 Project: Hadoop HDFS Issue Type:

[jira] [Updated] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11714: -- Attachment: HDFS-11714.v3.trunk.patch HDFS-11714.v3.branch-2.patch Attaching new patches

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991420#comment-15991420 ] Kihwal Lee commented on HDFS-11714: --- I will file a jira for the retention manager improv

[jira] [Updated] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11609: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.1 3.0.0-alpha3

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991339#comment-15991339 ] Kihwal Lee commented on HDFS-11714: --- bq. What if a VERSION file already exists in the di

[jira] [Commented] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991104#comment-15991104 ] Kihwal Lee commented on HDFS-11609: --- Only updated the comment in the new patch. Test fai

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-05-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991045#comment-15991045 ] Kihwal Lee commented on HDFS-11714: --- These are passing. {noformat}

[jira] [Updated] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11714: -- Attachment: HDFS-11714.v2.trunk.patch HDFS-11714.v2.branch-2.patch Attaching updated pat

[jira] [Updated] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11714: -- Target Version/s: 2.7.4, 2.8.1 (was: 2.8.1) > Newly added NN storage directory won't get initialized an

[jira] [Updated] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11609: -- Attachment: HDFS-11609_v3.trunk.patch HDFS-11609_v3.branch-2.patch HDFS-1

[jira] [Assigned] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-11714: - Assignee: Kihwal Lee > Newly added NN storage directory won't get initialized and cause space >

[jira] [Commented] (HDFS-11396) TestNameNodeMetadataConsistency#testGenerationStampInFuture timed out

2017-04-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988813#comment-15988813 ] Kihwal Lee commented on HDFS-11396: --- It could be due to interactions with other tests in

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988809#comment-15988809 ] Kihwal Lee commented on HDFS-11714: --- None of the findbugs warning was introduced by the

[jira] [Updated] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11714: -- Status: Patch Available (was: Open) > Newly added NN storage directory won't get initialized and cause

[jira] [Updated] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11714: -- Attachment: HDFS-11714.trunk.patch The version file can be created in an empty storage diectory in other

[jira] [Commented] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987756#comment-15987756 ] Kihwal Lee commented on HDFS-11714: --- The original design pre-HA was to just create direc

[jira] [Created] (HDFS-11714) Newly added NN storage directory won't get initialized and cause space exhaustion

2017-04-27 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11714: - Summary: Newly added NN storage directory won't get initialized and cause space exhaustion Key: HDFS-11714 URL: https://issues.apache.org/jira/browse/HDFS-11714 Project: Ha

[jira] [Commented] (HDFS-11709) StandbyCheckpointer should handle an non-existing legacyOivImageDir gracefully

2017-04-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986836#comment-15986836 ] Kihwal Lee commented on HDFS-11709: --- +1 lgtm > StandbyCheckpointer should handle an non

[jira] [Commented] (HDFS-5042) Completed files lost after power failure

2017-04-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985460#comment-15985460 ] Kihwal Lee commented on HDFS-5042: -- We've seen that too. Unless there is other bug in Hado

[jira] [Commented] (HDFS-10495) Block should be marked as missing if the all the replicas are on Decommissioned nodes.

2017-04-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985457#comment-15985457 ] Kihwal Lee commented on HDFS-10495: --- I've analyzed this and discussed potential solution

[jira] [Commented] (HDFS-5042) Completed files lost after power failure

2017-04-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984824#comment-15984824 ] Kihwal Lee commented on HDFS-5042: -- bq. Can we recover this kind of block from namenode af

[jira] [Assigned] (HDFS-7541) Upgrade Domains in HDFS

2017-04-25 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-7541: Assignee: Kihwal Lee (was: Ming Ma) > Upgrade Domains in HDFS > --- > >

[jira] [Commented] (HDFS-7541) Upgrade Domains in HDFS

2017-04-24 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981993#comment-15981993 ] Kihwal Lee commented on HDFS-7541: -- [~mingma] Since you guys have been using it in product

[jira] [Commented] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-24 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981295#comment-15981295 ] Kihwal Lee commented on HDFS-11691: --- bq. this way datanode link field on UI will also di

[jira] [Assigned] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-11691: - Assignee: Kihwal Lee > Add a proper scheme to the datanode links in NN web UI > -

[jira] [Updated] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11691: -- Status: Patch Available (was: Open) > Add a proper scheme to the datanode links in NN web UI >

[jira] [Updated] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11691: -- Attachment: HDFS-11691.patch > Add a proper scheme to the datanode links in NN web UI >

[jira] [Updated] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11691: -- Description: On the datanodes page of the namenode web UI, the datanode links may not be correct if the

[jira] [Updated] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11691: -- Description: On the datanodes page of the namenode web UI, the datanode links may not be correct if the

[jira] [Created] (HDFS-11691) Add a proper scheme to the datanode links in NN web UI

2017-04-21 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11691: - Summary: Add a proper scheme to the datanode links in NN web UI Key: HDFS-11691 URL: https://issues.apache.org/jira/browse/HDFS-11691 Project: Hadoop HDFS Issue Ty

[jira] [Comment Edited] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978923#comment-15978923 ] Kihwal Lee edited comment on HDFS-9922 at 4/21/17 3:58 PM: --- Don't

[jira] [Commented] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes

2017-04-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978923#comment-15978923 ] Kihwal Lee commented on HDFS-9922: -- Don't we need this in 2.8? > Upgrade Domain placement

[jira] [Commented] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-20 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977517#comment-15977517 ] Kihwal Lee commented on HDFS-11609: --- [~jojochuang], would you look at the updated patch?

[jira] [Commented] (HDFS-11661) GetContentSummary uses excessive amounts of memory

2017-04-19 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974623#comment-15974623 ] Kihwal Lee commented on HDFS-11661: --- [~xiaochen], [~jingzhao], any thoughts? We've inte

[jira] [Commented] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-18 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973539#comment-15973539 ] Kihwal Lee commented on HDFS-11609: --- Unit test failures are not related to the patch. I

[jira] [Updated] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-18 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11609: -- Attachment: HDFS-11609_v2.trunk.patch HDFS-11609_v2.branch-2.patch Attaching the updated

[jira] [Commented] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15971133#comment-15971133 ] Kihwal Lee commented on HDFS-11609: --- Agree it is a blocker. I will update the patch tod

[jira] [Updated] (HDFS-11648) Lazy construct the IIP pathname

2017-04-12 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11648: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.1 3.0.0-alpha3

<    2   3   4   5   6   7   8   9   10   11   >