[jira] [Updated] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11609: -- Status: Patch Available (was: Open) > Some blocks can be permanently lost if nodes are decommissioned

[jira] [Updated] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11609: -- Attachment: HDFS-11609.trunk.patch > Some blocks can be permanently lost if nodes are decommissioned

[jira] [Commented] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-04-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954245#comment-15954245 ] Kihwal Lee commented on HDFS-11609: --- bq. I have a patch with 1) and a unit test, but want to hear from

[jira] [Commented] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-03-31 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951587#comment-15951587 ] Kihwal Lee commented on HDFS-11609: --- h3. Inability to correctly guess the previous replication priority

[jira] [Assigned] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-03-31 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-11609: - Assignee: Kihwal Lee > Some blocks can be permanently lost if nodes are decommissioned while

[jira] [Created] (HDFS-11609) Some blocks can be permanently lost if nodes are decommissioned while dead

2017-03-31 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11609: - Summary: Some blocks can be permanently lost if nodes are decommissioned while dead Key: HDFS-11609 URL: https://issues.apache.org/jira/browse/HDFS-11609 Project: Hadoop

[jira] [Updated] (HDFS-11592) Closing a file has a wasteful preconditions

2017-03-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11592: -- Target Version/s: 2.8.1 > Closing a file has a wasteful preconditions >

[jira] [Updated] (HDFS-10601) Improve log message to include hostname when the NameNode is in safemode

2017-03-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-10601: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha3

[jira] [Commented] (HDFS-10601) Improve log message to include hostname when the NameNode is in safemode

2017-03-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928313#comment-15928313 ] Kihwal Lee commented on HDFS-10601: --- +1 looks good. > Improve log message to include hostname when the

[jira] [Commented] (HDFS-11417) Add datanode admin command to get the storage info.

2017-02-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868417#comment-15868417 ] Kihwal Lee commented on HDFS-11417: --- Sorry, it is actually you who are working on both issues! > Add

[jira] [Commented] (HDFS-11417) Add datanode admin command to get the storage info.

2017-02-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868413#comment-15868413 ] Kihwal Lee commented on HDFS-11417: --- More info was added to the datanoed UI. HDFS-11375 is improving it.

[jira] [Commented] (HDFS-11379) DFSInputStream may infinite loop requesting block locations

2017-02-13 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863850#comment-15863850 ] Kihwal Lee commented on HDFS-11379: --- The impact of this bug can vary depending on how wide the job is,

[jira] [Commented] (HDFS-11379) DFSInputStream may infinite loop requesting block locations

2017-02-13 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863840#comment-15863840 ] Kihwal Lee commented on HDFS-11379: --- Thanks, I just cherry-picked it to branch-2.8.0. > DFSInputStream

[jira] [Updated] (HDFS-11379) DFSInputStream may infinite loop requesting block locations

2017-02-13 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11379: -- Fix Version/s: (was: 2.8.1) 2.8.0 > DFSInputStream may infinite loop requesting

[jira] [Updated] (HDFS-11379) DFSInputStream may infinite loop requesting block locations

2017-02-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11379: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.1

[jira] [Commented] (HDFS-11379) DFSInputStream may infinite loop requesting block locations

2017-02-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861596#comment-15861596 ] Kihwal Lee commented on HDFS-11379: --- Nicely done. Although the retry in {{readWithStrategy()}} makes it

[jira] [Commented] (HDFS-11379) DFSInputStream may infinite loop requesting block locations

2017-02-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859820#comment-15859820 ] Kihwal Lee commented on HDFS-11379: --- I will review it today. > DFSInputStream may infinite loop

[jira] [Commented] (HDFS-11192) OOM during Quota Initialization lead to Namenode hang

2017-01-20 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832211#comment-15832211 ] Kihwal Lee commented on HDFS-11192: --- Looking at the java source, {{Thread.start0()}} is mapped to a JVM

[jira] [Commented] (HDFS-11192) OOM during Quota Initialization lead to Namenode hang

2017-01-20 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831904#comment-15831904 ] Kihwal Lee commented on HDFS-11192: --- We can create the quota init ForkJoinPool with a

[jira] [Commented] (HDFS-9574) Reduce client failures during datanode restart

2017-01-18 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828340#comment-15828340 ] Kihwal Lee commented on HDFS-9574: -- [~arpitagarwal], please elaborate the reason for marking it

[jira] [Commented] (HDFS-9409) DataNode shutdown does not guarantee full shutdown of all threads due to race condition.

2017-01-11 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819152#comment-15819152 ] Kihwal Lee commented on HDFS-9409: -- {{BlockScanner}} is the same way. It removes a thread from its

[jira] [Commented] (HDFS-7967) Reduce the performance impact of the balancer

2017-01-11 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818861#comment-15818861 ] Kihwal Lee commented on HDFS-7967: -- The 2.8 patch doesn't seem right. > Reduce the performance impact of

[jira] [Commented] (HDFS-3618) SSH fencing option may incorrectly succeed if nc (netcat) command not present

2017-01-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816250#comment-15816250 ] Kihwal Lee commented on HDFS-3618: -- The "-z" option is supported by the plain old BSD implementation of

[jira] [Commented] (HDFS-7967) Reduce the performance impact of the balancer

2017-01-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815227#comment-15815227 ] Kihwal Lee commented on HDFS-7967: -- I have reviewed the patch and am fine with it. We have used a variant

[jira] [Commented] (HDFS-9083) Replication violates block placement policy.

2017-01-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802662#comment-15802662 ] Kihwal Lee commented on HDFS-9083: -- [~djp], that's correct. > Replication violates block placement

[jira] [Commented] (HDFS-7967) Reduce the performance impact of the balancer

2017-01-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802511#comment-15802511 ] Kihwal Lee commented on HDFS-7967: -- The 2.8 patch works fine with jdk8. The unit tests ran fine too.

[jira] [Commented] (HDFS-11160) VolumeScanner reports write-in-progress replicas as corrupt incorrectly

2017-01-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801541#comment-15801541 ] Kihwal Lee commented on HDFS-11160: --- [~jojochuang], please make sure it is in 2.8. > VolumeScanner

[jira] [Commented] (HDFS-7784) load fsimage in parallel

2017-01-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795396#comment-15795396 ] Kihwal Lee commented on HDFS-7784: -- [~xiegang112], when you have a chance to test the performance, please

[jira] [Commented] (HDFS-11094) Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter

2016-12-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751772#comment-15751772 ] Kihwal Lee commented on HDFS-11094: --- I am fine with the latest patch. > Send back HAState along with

[jira] [Resolved] (HDFS-11207) Unnecessary incompatible change of NNHAStatusHeartbeat.state in DatanodeProtocolProtos

2016-12-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-11207. --- Resolution: Done > Unnecessary incompatible change of NNHAStatusHeartbeat.state in >

[jira] [Commented] (HDFS-11207) Unnecessary incompatible change of NNHAStatusHeartbeat.state in DatanodeProtocolProtos

2016-12-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733496#comment-15733496 ] Kihwal Lee commented on HDFS-11207: --- Reverted HDFS-5079 and HDFS-11094. > Unnecessary incompatible

[jira] [Reopened] (HDFS-11094) Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter

2016-12-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reopened HDFS-11094: --- I've reverted this, as HDFS-5079 was reverted. Please rework the patch to reflect that. > Send back

[jira] [Updated] (HDFS-5079) Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos.

2016-12-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5079: - Release Note: This change affects wire-compatibility of the NameNode/DataNode heartbeat protocol. Only

[jira] [Commented] (HDFS-5079) Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos.

2016-12-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733485#comment-15733485 ] Kihwal Lee commented on HDFS-5079: -- Reverted and updated the release note. > Cleaning up

[jira] [Updated] (HDFS-5079) Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos.

2016-12-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5079: - Fix Version/s: 3.0.0-alpha1 > Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos. >

[jira] [Updated] (HDFS-5079) Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos.

2016-12-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5079: - Fix Version/s: (was: 3.0.0-alpha1) > Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos. >

[jira] [Commented] (HDFS-5079) Cleaning up NNHAStatusHeartbeat.State DatanodeProtocolProtos.

2016-12-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732634#comment-15732634 ] Kihwal Lee commented on HDFS-5079: -- This doesn't seem to much value while breaking compatibility during

[jira] [Commented] (HDFS-11207) Unnecessary incompatible change of NNHAStatusHeartbeat.state in DatanodeProtocolProtos

2016-12-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732628#comment-15732628 ] Kihwal Lee commented on HDFS-11207: --- I am going to revert HDFS-5079 later today if there is no

[jira] [Comment Edited] (HDFS-11160) VolumeScanner reports write-in-progress replicas as corrupt incorrectly

2016-12-07 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729073#comment-15729073 ] Kihwal Lee edited comment on HDFS-11160 at 12/7/16 3:53 PM: +1 The patch looks

[jira] [Commented] (HDFS-11160) VolumeScanner reports write-in-progress replicas as corrupt incorrectly

2016-12-07 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729073#comment-15729073 ] Kihwal Lee commented on HDFS-11160: --- The patch looks good. About {noformat} // TODO: we only need to

[jira] [Commented] (HDFS-11207) Unnecessary incompatible change of NNHAStatusHeartbeat.state in DatanodeProtocolProtos

2016-12-06 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726418#comment-15726418 ] Kihwal Lee commented on HDFS-11207: --- I think we should revert HDFS-5079. It breaks compatibility while

[jira] [Updated] (HDFS-11198) NN UI should link DN web address using hostnames

2016-12-06 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11198: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2

[jira] [Commented] (HDFS-11198) NN UI should link DN web address using hostnames

2016-12-06 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15725570#comment-15725570 ] Kihwal Lee commented on HDFS-11198: --- +1 > NN UI should link DN web address using hostnames >

[jira] [Commented] (HDFS-11198) NN UI should link DN web address using hostnames

2016-12-05 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722401#comment-15722401 ] Kihwal Lee commented on HDFS-11198: --- The build master is down at this moment. > NN UI should link DN

[jira] [Commented] (HDFS-11094) Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter

2016-12-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716687#comment-15716687 ] Kihwal Lee commented on HDFS-11094: --- bq. For the unit test, can we set a very large heartbeat interval

[jira] [Commented] (HDFS-11146) Excess replicas will not be deleted until all storages's FBR received after failover

2016-12-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715608#comment-15715608 ] Kihwal Lee commented on HDFS-11146: --- Sorry, I am busy and won't be able to review it properly soon. I

[jira] [Updated] (HDFS-11198) NN UI should link DN web address using hostnames

2016-12-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11198: -- Assignee: Weiwei Yang > NN UI should link DN web address using hostnames >

[jira] [Commented] (HDFS-11198) NN UI should link DN web address using hostnames

2016-12-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715214#comment-15715214 ] Kihwal Lee commented on HDFS-11198: --- sure go ahead. > NN UI should link DN web address using hostnames

[jira] [Updated] (HDFS-8674) Improve performance of postponed block scans

2016-12-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8674: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.8.0

[jira] [Updated] (HDFS-11198) NN UI should link DN web address using hostnames

2016-12-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11198: -- Description: The new NN UI shows links to DN web pages, but since the link is from the info address

[jira] [Commented] (HDFS-8674) Improve performance of postponed block scans

2016-12-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712633#comment-15712633 ] Kihwal Lee commented on HDFS-8674: -- +1 The patch looks good. > Improve performance of postponed block

[jira] [Created] (HDFS-11198) NN UI should link DN web address using hostnames

2016-12-01 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11198: - Summary: NN UI should link DN web address using hostnames Key: HDFS-11198 URL: https://issues.apache.org/jira/browse/HDFS-11198 Project: Hadoop HDFS Issue Type:

[jira] [Commented] (HDFS-8674) Improve performance of postponed block scans

2016-12-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712103#comment-15712103 ] Kihwal Lee commented on HDFS-8674: -- {{LightWeightLinkedSet}} does not support {{remove()}}. > Improve

[jira] [Commented] (HDFS-11131) TestThrottledAsyncChecker#testContextIsPassed is flaky

2016-12-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712070#comment-15712070 ] Kihwal Lee commented on HDFS-11131: --- {{testCancellation}} also fails. Failure doesn't seem to be rare.

[jira] [Commented] (HDFS-11179) LightWeightHashSet can't remove blocks correctly which have a large number blockId

2016-12-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712026#comment-15712026 ] Kihwal Lee commented on HDFS-11179: --- We have been running with HDFS-8674 internally for quite some time

[jira] [Commented] (HDFS-11192) OOM during Quota Initialization lead to Namenode hang

2016-11-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710722#comment-15710722 ] Kihwal Lee commented on HDFS-11192: --- Was it hitting its ulimit on # of threads/processes? If that is

[jira] [Commented] (HDFS-8674) Improve performance of postponed block scans

2016-11-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709744#comment-15709744 ] Kihwal Lee commented on HDFS-8674: -- Looks like {{HDFS-8674.2.patch}} is for branch-2. Can you post a

[jira] [Updated] (HDFS-8674) Improve performance of postponed block scans

2016-11-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8674: - Target Version/s: 2.8.0, 3.0.0-alpha2 (was: 2.8.0, 2.7.4, 3.0.0-alpha2) > Improve performance of

[jira] [Commented] (HDFS-11180) Intermittent deadlock in NameNode when failover happens.

2016-11-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709732#comment-15709732 ] Kihwal Lee commented on HDFS-11180: --- +1 for the patch. I am not sure whether it is worth mentioning the

[jira] [Commented] (HDFS-11180) Intermittent deadlock in NameNode when failover happens.

2016-11-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702354#comment-15702354 ] Kihwal Lee commented on HDFS-11180: --- bq. NameNode holds a lock of FSEditLog and requires a lock of

[jira] [Commented] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode

2016-11-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685014#comment-15685014 ] Kihwal Lee commented on HDFS-10966: --- Committed to branch-2.7. Thanks for working on this [~mwagner] and

[jira] [Updated] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode

2016-11-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-10966: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2

[jira] [Commented] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode

2016-11-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685011#comment-15685011 ] Kihwal Lee commented on HDFS-10966: --- The 2.7 patch looks good. > Enhance Dispatcher logic on deciding

[jira] [Commented] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode

2016-11-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684079#comment-15684079 ] Kihwal Lee commented on HDFS-10966: --- I've committed this to trunk through branch-2.8. Please provide a

[jira] [Commented] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode

2016-11-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15683978#comment-15683978 ] Kihwal Lee commented on HDFS-10966: --- +1 > Enhance Dispatcher logic on deciding when to give up a source

[jira] [Commented] (HDFS-8824) Do not use small blocks for balancing the cluster

2016-11-18 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677540#comment-15677540 ] Kihwal Lee commented on HDFS-8824: -- While this will initially increase the efficiency of balancing, it is

[jira] [Commented] (HDFS-11146) Excess replicas will not be deleted until all storages's FBR received after failover

2016-11-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673862#comment-15673862 ] Kihwal Lee commented on HDFS-11146: --- NN could tell datanodes to send FBR in a heartbeat response. This

[jira] [Commented] (HDFS-11153) RPC Client detect address changed should reconnect immediately

2016-11-17 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673846#comment-15673846 ] Kihwal Lee commented on HDFS-11153: --- You can enable this through config. Here is an example: {noformat}

[jira] [Commented] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode

2016-11-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671089#comment-15671089 ] Kihwal Lee commented on HDFS-10966: --- the patch looks reasonable in general. - It is replacing the logic

[jira] [Comment Edited] (HDFS-8870) Lease is leaked on write failure

2016-11-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668304#comment-15668304 ] Kihwal Lee edited comment on HDFS-8870 at 11/15/16 9:02 PM: Committed this to

[jira] [Commented] (HDFS-8870) Lease is leaked on write failure

2016-11-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668304#comment-15668304 ] Kihwal Lee commented on HDFS-8870: -- Committed this to trunk, branch-2 and branch-2.8. > Lease is leaked

[jira] [Updated] (HDFS-8870) Lease is leaked on write failure

2016-11-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8870: - Fix Version/s: 3.0.0-alpha2 2.8.0 > Lease is leaked on write failure >

[jira] [Updated] (HDFS-8870) Lease is leaked on write failure

2016-11-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-8870: - Assignee: Kuhu Shukla (was: Daryn Sharp) > Lease is leaked on write failure >

[jira] [Commented] (HDFS-8870) Lease is leaked on write failure

2016-11-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668242#comment-15668242 ] Kihwal Lee commented on HDFS-8870: -- +1 The patch looks good. > Lease is leaked on write failure >

[jira] [Commented] (HDFS-11056) Concurrent append and read operations lead to checksum error

2016-11-11 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657402#comment-15657402 ] Kihwal Lee commented on HDFS-11056: --- +1 for the 2.7 patch. It looks to be a correct port. Thanks

[jira] [Commented] (HDFS-11114) Support for running async disk checks in DataNode

2016-11-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651285#comment-15651285 ] Kihwal Lee commented on HDFS-4: --- This failed recently.

[jira] [Commented] (HDFS-11115) Remove bytes2Array and string2Bytes

2016-11-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648021#comment-15648021 ] Kihwal Lee commented on HDFS-5: --- You are undoing this optimization. {code} // Using the charset

[jira] [Commented] (HDFS-11114) Support for running async disk checks in DataNode

2016-11-08 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15647937#comment-15647937 ] Kihwal Lee commented on HDFS-4: --- It broke branch-2. [~steve_l] also noticed it. > Support for

[jira] [Commented] (HDFS-11056) Concurrent append and read operations lead to checksum error

2016-11-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633249#comment-15633249 ] Kihwal Lee commented on HDFS-11056: --- I was looking at the patch since yesterday. It looks like the

[jira] [Commented] (HDFS-11090) Leave safemode immediately if all blocks have reported in

2016-11-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630287#comment-15630287 ] Kihwal Lee commented on HDFS-11090: --- I agree with [~jingzhao]. If people don't want safe mode extension,

[jira] [Commented] (HDFS-11078) NPE in LazyPersistFileScrubber

2016-11-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625742#comment-15625742 ] Kihwal Lee commented on HDFS-11078: --- Since this can happen legitimately and does not require any further

[jira] [Updated] (HDFS-10997) Reduce number of path resolving methods

2016-11-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-10997: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved

[jira] [Commented] (HDFS-10997) Reduce number of path resolving methods

2016-11-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625365#comment-15625365 ] Kihwal Lee commented on HDFS-10997: --- The branch-2 patch looks good. I will commit it shortly. > Reduce

[jira] [Updated] (HDFS-11069) Tighten the authorization of datanode RPC

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11069: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2

[jira] [Commented] (HDFS-11069) Tighten the authorization of datanode RPC

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612896#comment-15612896 ] Kihwal Lee commented on HDFS-11069: --- Committed to trunk, branch-2, branch-2.8 and branch-2.7. > Tighten

[jira] [Commented] (HDFS-11069) Tighten the authorization of datanode RPC

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612863#comment-15612863 ] Kihwal Lee commented on HDFS-11069: --- Thanks for the review, Daryn. I've verified it working as expected

[jira] [Reopened] (HDFS-10455) Logging the username when deny the setOwner operation

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reopened HDFS-10455: --- Reverted the commits. > Logging the username when deny the setOwner operation >

[jira] [Updated] (HDFS-10455) Logging the username when deny the setOwner operation

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-10455: -- Fix Version/s: (was: 3.0.0-alpha2) (was: 2.8.0) > Logging the username when

[jira] [Commented] (HDFS-11069) Tighten the authorization of datanode RPC

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612831#comment-15612831 ] Kihwal Lee commented on HDFS-11069: --- {{TestPermission}} is broken by HDFS-10455. The other test passes.

[jira] [Commented] (HDFS-10455) Logging the username when deny the setOwner operation

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612820#comment-15612820 ] Kihwal Lee commented on HDFS-10455: --- The precommit was from May. You should have re-run it. > Logging

[jira] [Commented] (HDFS-10455) Logging the username when deny the setOwner operation

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612810#comment-15612810 ] Kihwal Lee commented on HDFS-10455: --- It broke {{TestPermission}} {noformat} Running

[jira] [Commented] (HDFS-11063) Set NameNode RPC server handler thread name with more descriptive information about the RPC call.

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612485#comment-15612485 ] Kihwal Lee commented on HDFS-11063: --- Once side-effect is that the namenode log can get bigger and

[jira] [Updated] (HDFS-11069) Tighten the authorization of datanode RPC

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11069: -- Attachment: HDFS-11069.patch > Tighten the authorization of datanode RPC >

[jira] [Updated] (HDFS-11069) Tighten the authorization of datanode RPC

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11069: -- Status: Patch Available (was: Open) > Tighten the authorization of datanode RPC >

[jira] [Updated] (HDFS-11069) Tighten the authorization of datanode RPC

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11069: -- Description: The current implementation of {{checkSuperuserPrivilege()}} allows the datanode user from

[jira] [Commented] (HDFS-11069) Tighten the authorization of datanode RPC

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612304#comment-15612304 ] Kihwal Lee commented on HDFS-11069: --- {code} private void checkSuperuserPrivilege() { ... if

[jira] [Assigned] (HDFS-11069) Tighten the authorization of datanode RPC

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-11069: - Assignee: Kihwal Lee > Tighten the authorization of datanode RPC >

[jira] [Updated] (HDFS-11069) Tighten the authorization of datanode RPC

2016-10-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11069: -- Issue Type: Improvement (was: Bug) > Tighten the authorization of datanode RPC >

[jira] [Created] (HDFS-11069) Tighten the authorization of datanode RPC

2016-10-27 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-11069: - Summary: Tighten the authorization of datanode RPC Key: HDFS-11069 URL: https://issues.apache.org/jira/browse/HDFS-11069 Project: Hadoop HDFS Issue Type: Bug

[jira] [Updated] (HDFS-11053) Unnecessary superuser check in versionRequest()

2016-10-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11053: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2

<    3   4   5   6   7   8   9   10   11   12   >