[jira] [Commented] (HDFS-7613) Block placement policy for erasure coding groups

2015-07-15 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629070#comment-14629070 ] Ming Ma commented on HDFS-7613: --- Interesting work. There are couple issues around the

[jira] [Commented] (HDFS-8776) Decom manager should not be active on standby

2015-07-15 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629236#comment-14629236 ] Ming Ma commented on HDFS-8776: --- Make sense. There might be some operational impact with

[jira] [Commented] (HDFS-8670) Better to exclude decommissioned nodes for namenode NodeUsage JMX

2015-07-15 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629217#comment-14629217 ] Ming Ma commented on HDFS-8670: --- Thanks [~andreina]. Overall it looks good. The two test

[jira] [Updated] (HDFS-7314) When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClient

2015-07-13 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7314: -- Attachment: HDFS-7314-9.patch Thanks [~cmccabe]. Here is the update patch with your suggestion. I will open a

[jira] [Updated] (HDFS-7314) Aborted DFSClient's impact on long running service like YARN

2015-07-08 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7314: -- Attachment: HDFS-7314-8.patch Here is a slightly different version we have deployed on our production clusters.

[jira] [Commented] (HDFS-8670) Better to exclude decommissioned nodes for namenode NodeUsage JMX

2015-07-08 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619713#comment-14619713 ] Ming Ma commented on HDFS-8670: --- Thanks [~andreina] for the patch! * Maybe we should exclude

[jira] [Commented] (HDFS-8690) LeaseRenewer should not abort DFSClient when renew fails

2015-07-06 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615732#comment-14615732 ] Ming Ma commented on HDFS-8690: --- We actually deployed a slightly modified version of

[jira] [Commented] (HDFS-8690) LeaseRenewer should not abort DFSClient when renew fails

2015-06-29 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14606006#comment-14606006 ] Ming Ma commented on HDFS-8690: --- Thanks, [~lichangleo] [~arpitagarwal]. There were some

[jira] [Commented] (HDFS-8656) Preserve compatibility of ClientProtocol#rollingUpgrade after finalization

2015-06-26 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603138#comment-14603138 ] Ming Ma commented on HDFS-8656: --- +1. Thanks Andrew. The unit test failures are unrelated.

[jira] [Commented] (HDFS-8674) Improve performance of postponed block scans

2015-06-26 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603873#comment-14603873 ] Ming Ma commented on HDFS-8674: --- Thanks [~daryn]! * The motivation of randomization is to

[jira] [Commented] (HDFS-8656) Preserve compatibility of ClientProtocol#rollingUpgrade after finalization

2015-06-25 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14602043#comment-14602043 ] Ming Ma commented on HDFS-8656: --- Thanks, Andrew. Latest patch looks good. Nit: since you are

[jira] [Created] (HDFS-8670) Better to exclude decommissioned nodes for namenode NodeUsage JMX

2015-06-25 Thread Ming Ma (JIRA)
Ming Ma created HDFS-8670: - Summary: Better to exclude decommissioned nodes for namenode NodeUsage JMX Key: HDFS-8670 URL: https://issues.apache.org/jira/browse/HDFS-8670 Project: Hadoop HDFS Issue

[jira] [Commented] (HDFS-8656) Preserve compatibility of ClientProtocol#rollingUpgrade after finalization

2015-06-24 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600272#comment-14600272 ] Ming Ma commented on HDFS-8656: --- To clarify what I meant. If the plan is still to return null

[jira] [Commented] (HDFS-1431) Balancer should work with the logic of BlockPlacementPolicy

2015-06-22 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14597133#comment-14597133 ] Ming Ma commented on HDFS-1431: --- When I met with [~andrew.wang], [~ctrezzo], [~atm],

[jira] [Commented] (HDFS-7214) Display the time when NN became active on the webUI

2015-06-22 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596981#comment-14596981 ] Ming Ma commented on HDFS-7214: --- [~l201514] you are right that HDFS-7257 has added the

[jira] [Created] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy

2015-06-22 Thread Ming Ma (JIRA)
Ming Ma created HDFS-8647: - Summary: Abstract BlockManager's rack policy into BlockPlacementPolicy Key: HDFS-8647 URL: https://issues.apache.org/jira/browse/HDFS-8647 Project: Hadoop HDFS Issue

[jira] [Updated] (HDFS-7541) Upgrade Domains in HDFS

2015-06-19 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7541: -- Attachment: HDFS-7541-2.patch Upgrade Domains in HDFS --- Key: HDFS-7541

[jira] [Commented] (HDFS-7877) Support maintenance state for datanodes

2015-06-19 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594075#comment-14594075 ] Ming Ma commented on HDFS-7877: --- Thanks [~rajive] for your input! I also discussed with

[jira] [Updated] (HDFS-7541) Upgrade Domains in HDFS

2015-06-19 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7541: -- Attachment: (was: HDFS-7541-3.patch) Upgrade Domains in HDFS ---

[jira] [Updated] (HDFS-7541) Upgrade Domains in HDFS

2015-06-19 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7541: -- Attachment: HDFS-7541-3.patch We have been running upgrade domain policy on one of our large production

[jira] [Commented] (HDFS-7609) Avoid retry cache collision when Standby NameNode loading edits

2015-05-29 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565353#comment-14565353 ] Ming Ma commented on HDFS-7609: --- Thanks Jing and all other folks. Avoid retry cache

[jira] [Updated] (HDFS-7609) startup used too much time to load edits

2015-05-28 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7609: -- Attachment: HDFS-7609-3.patch Jing, I agree with you. Here is the updated patch. startup used too much time to

[jira] [Commented] (HDFS-7609) startup used too much time to load edits

2015-05-27 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562214#comment-14562214 ] Ming Ma commented on HDFS-7609: --- Thanks [~jingzhao]. Good point about saveNamespace.

[jira] [Updated] (HDFS-7609) startup used too much time to load edits

2015-05-26 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7609: -- Attachment: HDFS-7609-2.patch [~jingzhao] you are right. It is possible that nn starts the transition right

[jira] [Commented] (HDFS-7609) startup used too much time to load edits

2015-05-21 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14555003#comment-14555003 ] Ming Ma commented on HDFS-7609: --- Thanks, [~jingzhao]. In the scenario you described, IIUC, in

[jira] [Updated] (HDFS-6184) Capture NN's thread dump when it fails over

2015-05-12 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-6184: -- Attachment: HDFS-6184-5.patch Thanks [~ajisakaa]. Here is the new patch. Capture NN's thread dump when it

[jira] [Updated] (HDFS-6184) Capture NN's thread dump when it fails over

2015-05-12 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-6184: -- Attachment: HDFS-6184-6.patch New patch to fix the whilespace issue. The TestHdfsConfigFields failure is due to

[jira] [Commented] (HDFS-6184) Capture NN's thread dump when it fails over

2015-05-12 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541242#comment-14541242 ] Ming Ma commented on HDFS-6184: --- Thanks [~ajisakaa] and others. Capture NN's thread dump

[jira] [Updated] (HDFS-7609) startup used too much time to load edits

2015-05-08 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7609: -- Labels: BB2015-05-RFC (was: BB2015-05-TBR) startup used too much time to load edits

[jira] [Updated] (HDFS-8056) Decommissioned dead nodes should continue to be counted as dead after NN restart

2015-05-08 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8056: -- Labels: BB2015-05-RFC (was: BB2015-05-TBR) Decommissioned dead nodes should continue to be counted as dead

[jira] [Updated] (HDFS-6184) Capture NN's thread dump when it fails over

2015-05-07 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-6184: -- Attachment: HDFS-6184-3.patch Capture NN's thread dump when it fails over

[jira] [Updated] (HDFS-6184) Capture NN's thread dump when it fails over

2015-05-07 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-6184: -- Labels: BB2015-05-RFC (was: BB2015-05-TBR) Capture NN's thread dump when it fails over

[jira] [Updated] (HDFS-6184) Capture NN's thread dump when it fails over

2015-05-07 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-6184: -- Attachment: HDFS-6184-4.patch Thanks [~ajisakaa]. Here is the updated patch with your suggestions. I also

[jira] [Updated] (HDFS-7609) startup used too much time to load edits

2015-05-04 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7609: -- Attachment: HDFS-7609.patch Here is the draft patch that prevents client from polluting the retry cache when

[jira] [Updated] (HDFS-7609) startup used too much time to load edits

2015-05-04 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7609: -- Assignee: Ming Ma Status: Patch Available (was: Open) startup used too much time to load edits

[jira] [Commented] (HDFS-7609) startup used too much time to load edits

2015-05-04 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527030#comment-14527030 ] Ming Ma commented on HDFS-7609: --- #3 in the scenario description above should be Before nn2

[jira] [Updated] (HDFS-5757) refreshNodes with many nodes at the same time could slow down NN

2015-05-04 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-5757: -- Resolution: Duplicate Status: Resolved (was: Patch Available) This optimization has been taken care of

[jira] [Commented] (HDFS-6184) Capture NN's thread dump when it fails over

2015-05-04 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527042#comment-14527042 ] Ming Ma commented on HDFS-6184: --- Anyone gets time to review this? We have been running with

[jira] [Commented] (HDFS-7281) Missing block is marked as corrupted block

2015-05-01 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523402#comment-14523402 ] Ming Ma commented on HDFS-7281: --- Thanks [~yzhangal] and [~lohit], [~aw], [~cnauroth],

[jira] [Resolved] (HDFS-7442) Optimization for decommission-in-progress check

2015-05-01 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma resolved HDFS-7442. --- Resolution: Duplicate HDFS-7411 has addressed this issue. Optimization for decommission-in-progress check

[jira] [Commented] (HDFS-7609) startup used too much time to load edits

2015-05-01 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523562#comment-14523562 ] Ming Ma commented on HDFS-7609: --- Small retryCache size can also impact the correctness, given

[jira] [Commented] (HDFS-6697) Make NN lease soft and hard limits configurable

2015-05-01 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523589#comment-14523589 ] Ming Ma commented on HDFS-6697: --- Thanks, [~andreina]. When you are at it, it might be useful

[jira] [Updated] (HDFS-7281) Missing block is marked as corrupted block

2015-04-29 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7281: -- Attachment: HDFS-7281-6.patch Thanks [~yzhangal]. Here is the updated patch. Missing block is marked as

[jira] [Commented] (HDFS-8174) Update replication count to live rep count in fsck report

2015-04-29 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520453#comment-14520453 ] Ming Ma commented on HDFS-8174: --- Thanks [~andreina]. LGTM. Update replication count to live

[jira] [Commented] (HDFS-8056) Decommissioned dead nodes should continue to be counted as dead after NN restart

2015-04-28 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518357#comment-14518357 ] Ming Ma commented on HDFS-8056: --- [~andrew.wang] and others, appreciate any input you might

[jira] [Updated] (HDFS-7281) Missing block is marked as corrupted block

2015-04-28 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7281: -- Release Note: The patch improves the reporting around missing blocks and corrupted blocks. 1. A block is

[jira] [Updated] (HDFS-7281) Missing block is marked as corrupted block

2015-04-28 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7281: -- Attachment: HDFS-7281-5.patch [~yzhangal], good catch. I have updated the patch and the release notes. Missing

[jira] [Updated] (HDFS-7281) Missing block is marked as corrupted block

2015-04-20 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7281: -- Attachment: HDFS-7281-4.patch Yongjun, that makes sense. Here is the updated patch. Missing block is marked as

[jira] [Commented] (HDFS-7993) Incorrect descriptions in fsck when nodes are decommissioned

2015-04-17 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500482#comment-14500482 ] Ming Ma commented on HDFS-7993: --- Could the test fail if the node becomes decommissioned right

[jira] [Commented] (HDFS-7281) Missing block is marked as corrupted block

2015-04-17 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14501082#comment-14501082 ] Ming Ma commented on HDFS-7281: --- For the incompatibility issue Lohit mentioned, if some tools

[jira] [Updated] (HDFS-7281) Missing block is marked as corrupted block

2015-04-17 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7281: -- Attachment: HDFS-7281-3.patch Thanks, [~yzhangal]. Here is the rebased patch. Missing block is marked as

[jira] [Commented] (HDFS-7993) Incorrect descriptions in fsck when nodes are decommissioned

2015-04-16 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499111#comment-14499111 ] Ming Ma commented on HDFS-7993: --- bq. Can we do that in a separate JIRA? I am fine with that.

[jira] [Commented] (HDFS-7993) Incorrect descriptions in fsck when nodes are decommissioned

2015-04-14 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494372#comment-14494372 ] Ming Ma commented on HDFS-7993: --- Thanks, Vinay. bq. In the current output, blk_x len=y

[jira] [Commented] (HDFS-7993) Incorrect descriptions in fsck when nodes are decommissioned

2015-04-13 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492489#comment-14492489 ] Ming Ma commented on HDFS-7993: --- HDFS-7933 has improved the replica reporting in the case of

[jira] [Commented] (HDFS-7725) Incorrect nodes in service metrics caused all writes to fail

2015-04-13 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492492#comment-14492492 ] Ming Ma commented on HDFS-7725: --- Thanks Zhe and Andrew. Incorrect nodes in service metrics

[jira] [Commented] (HDFS-8025) Addendum fix for HDFS-3087 Decomissioning on NN restart can complete without blocks being replicated

2015-04-13 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492493#comment-14492493 ] Ming Ma commented on HDFS-8025: --- Thanks Andrew and Kihwal. Addendum fix for HDFS-3087

[jira] [Commented] (HDFS-8009) Signal congestion on the DataNode

2015-04-13 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492523#comment-14492523 ] Ming Ma commented on HDFS-8009: --- This overall DN congestion functionality seem useful. *

[jira] [Updated] (HDFS-8056) Decommissioned dead nodes should continue to be counted as dead after NN restart

2015-04-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8056: -- Attachment: HDFS-8056-2.patch Updated patch to fix test failure in TestHostFileManager. Decommissioned dead

[jira] [Updated] (HDFS-7725) Incorrect nodes in service metrics caused all writes to fail

2015-04-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7725: -- Attachment: HDFS-7725-3.patch Thanks, Andrew and Zhe. The latest patch moves the liveness check to

[jira] [Updated] (HDFS-7877) Support maintenance state for datanodes

2015-04-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7877: -- Attachment: Supportmaintenancestatefordatanodes-2.pdf HDFS-7877-2.patch Here is the updated

[jira] [Created] (HDFS-8056) Decommissioned dead nodes should continue to be counted as dead after NN restart

2015-04-02 Thread Ming Ma (JIRA)
Ming Ma created HDFS-8056: - Summary: Decommissioned dead nodes should continue to be counted as dead after NN restart Key: HDFS-8056 URL: https://issues.apache.org/jira/browse/HDFS-8056 Project: Hadoop HDFS

[jira] [Updated] (HDFS-8056) Decommissioned dead nodes should continue to be counted as dead after NN restart

2015-04-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8056: -- Attachment: HDFS-8056.patch Here is the initial patch. It put the dead node under (dead, decommissioned) after

[jira] [Updated] (HDFS-8056) Decommissioned dead nodes should continue to be counted as dead after NN restart

2015-04-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8056: -- Assignee: Ming Ma Status: Patch Available (was: Open) Decommissioned dead nodes should continue to be

[jira] [Updated] (HDFS-8025) Addendum for HDFS-3087

2015-04-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8025: -- Attachment: HDFS-8025-2.patch Thanks Andrew. Here is the updated patch with your suggestions. Addendum for

[jira] [Updated] (HDFS-8025) Addendum for HDFS-3087

2015-03-30 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8025: -- Assignee: Ming Ma Status: Patch Available (was: Open) Addendum for HDFS-3087 --

[jira] [Updated] (HDFS-8025) Addendum for HDFS-3087

2015-03-30 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-8025: -- Attachment: HDFS-8025.patch The reason unit test TestDecommission#testDecommissionWithNamenodeRestart can pass

[jira] [Created] (HDFS-8025) Addendum for HDFS-3087

2015-03-30 Thread Ming Ma (JIRA)
Ming Ma created HDFS-8025: - Summary: Addendum for HDFS-3087 Key: HDFS-8025 URL: https://issues.apache.org/jira/browse/HDFS-8025 Project: Hadoop HDFS Issue Type: Bug Reporter: Ming Ma

[jira] [Updated] (HDFS-7725) Incorrect nodes in service metrics caused all writes to fail

2015-03-30 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7725: -- Description: `One of our clusters sometimes couldn't allocate blocks from any DNs. BlockPlacementPolicyDefault

[jira] [Updated] (HDFS-7725) Incorrect nodes in service metrics caused all writes to fail

2015-03-30 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7725: -- Description: One of our clusters sometimes couldn't allocate blocks from any DNs. BlockPlacementPolicyDefault

[jira] [Updated] (HDFS-7725) Incorrect nodes in service metrics caused all writes to fail

2015-03-30 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7725: -- Attachment: HDFS-7725-2.patch Thanks, Zhe. Here is the rebase of the patch. Incorrect nodes in service metrics

[jira] [Commented] (HDFS-7411) Refactor and improve decommissioning logic into DecommissionManager

2015-03-27 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384569#comment-14384569 ] Ming Ma commented on HDFS-7411: --- [~andrew.wang], thanks for great work. Just to confirm, is

[jira] [Commented] (HDFS-7411) Refactor and improve decommissioning logic into DecommissionManager

2015-03-27 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384817#comment-14384817 ] Ming Ma commented on HDFS-7411: --- Thanks, Andrew. I will file a jira for it. Another issue is

[jira] [Commented] (HDFS-7993) Incorrect descriptions in fsck when nodes are decommissioned

2015-03-25 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14381305#comment-14381305 ] Ming Ma commented on HDFS-7993: --- Thanks, [~andreina]. Please go ahead. We might want to

[jira] [Updated] (HDFS-7993) Incorrect descriptions in fsck when nodes are decommissioned

2015-03-25 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7993: -- Description: When you run fsck with -files or -racks, you will get something like below if one of the replicas

[jira] [Created] (HDFS-7993) Incorrect descriptions in fsck when nodes are decommissioned

2015-03-25 Thread Ming Ma (JIRA)
Ming Ma created HDFS-7993: - Summary: Incorrect descriptions in fsck when nodes are decommissioned Key: HDFS-7993 URL: https://issues.apache.org/jira/browse/HDFS-7993 Project: Hadoop HDFS Issue

[jira] [Updated] (HDFS-7491) Add incremental blockreport latency to DN metrics

2015-03-12 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7491: -- Attachment: HDFS-7491-4.patch Thanks Xiaoyu for suggestion. Here is the updated patch. Chris, given

[jira] [Commented] (HDFS-7877) Support maintenance state for datanodes

2015-03-11 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356375#comment-14356375 ] Ming Ma commented on HDFS-7877: --- Thanks Eddy for the review and suggestions. Please find my

[jira] [Commented] (HDFS-7491) Add incremental blockreport latency to DN metrics

2015-03-11 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358133#comment-14358133 ] Ming Ma commented on HDFS-7491: --- Thanks Xiaoyu and Chris. Add incremental blockreport

[jira] [Commented] (HDFS-7877) Support maintenance state for datanodes

2015-03-08 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352471#comment-14352471 ] Ming Ma commented on HDFS-7877: --- Thanks Allen for pointing out. We didn't know about

[jira] [Commented] (HDFS-6729) Support maintenance mode for DN

2015-03-08 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352477#comment-14352477 ] Ming Ma commented on HDFS-6729: --- Eddy, thanks for the work. We didn't know about this at all

[jira] [Updated] (HDFS-7877) Support maintenance state for datanodes

2015-03-06 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7877: -- Attachment: HDFS-7877.patch Supportmaintenancestatefordatanodes.pdf Here are the initial design

[jira] [Created] (HDFS-7877) Support maintenance state for datanodes

2015-03-03 Thread Ming Ma (JIRA)
Ming Ma created HDFS-7877: - Summary: Support maintenance state for datanodes Key: HDFS-7877 URL: https://issues.apache.org/jira/browse/HDFS-7877 Project: Hadoop HDFS Issue Type: New Feature

[jira] [Updated] (HDFS-7541) Upgrade Domains in HDFS

2015-03-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7541: -- Attachment: (was: HDFS-7541.patch) Upgrade Domains in HDFS --- Key:

[jira] [Updated] (HDFS-7541) Upgrade Domains in HDFS

2015-03-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7541: -- Attachment: HDFS-7541.patch UpgradeDomains_design_v2.pdf Here are the updated design document

[jira] [Updated] (HDFS-7541) Upgrade Domains in HDFS

2015-03-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7541: -- Attachment: HDFS-7541.patch Upgrade Domains in HDFS --- Key: HDFS-7541

[jira] [Updated] (HDFS-7541) Upgrade Domains in HDFS

2015-03-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7541: -- Attachment: HDFS-7541.patch Upgrade Domains in HDFS --- Key: HDFS-7541

[jira] [Updated] (HDFS-7541) Upgrade Domains in HDFS

2015-03-03 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7541: -- Attachment: (was: HDFS-7541.patch) Upgrade Domains in HDFS --- Key:

[jira] [Commented] (HDFS-7009) Active NN and standby NN have different live nodes

2015-02-23 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334076#comment-14334076 ] Ming Ma commented on HDFS-7009: --- Thanks, Chris, Arpit and Nicholas. Active NN and standby

[jira] [Updated] (HDFS-7491) Add incremental blockreport latency to DN metrics

2015-02-20 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7491: -- Attachment: HDFS-7491-branch-2.patch HDFS-7491-3.patch Thanks, Chris. Here are the updated

[jira] [Updated] (HDFS-7009) Active NN and standby NN have different live nodes

2015-02-20 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7009: -- Attachment: HDFS-7009-4.patch Thanks, Chris. Here is the updated patch. Nicholas can confirm,

[jira] [Updated] (HDFS-7491) Add incremental blockreport latency to DN metrics

2015-02-17 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7491: -- Attachment: HDFS-7491-2.patch Thanks, Chris. Here is the updated patch to have the unit test verify the metrics

[jira] [Updated] (HDFS-7009) Active NN and standby NN have different live nodes

2015-02-14 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7009: -- Attachment: HDFS-7009-3.patch Thanks, Arpit. Yes, I meant reRegister. Thanks, Chris. I agree with both of your

[jira] [Commented] (HDFS-7009) Active NN and standby NN have different live nodes

2015-02-12 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319549#comment-14319549 ] Ming Ma commented on HDFS-7009: --- Thanks, [~arpitagarwal]. The patch seems to be useful even

[jira] [Commented] (HDFS-7725) Incorrect nodes in service metrics caused all writes to fail

2015-02-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302345#comment-14302345 ] Ming Ma commented on HDFS-7725: --- Thanks, [~zhz]. Yes, the logic of don't modify nn stats if

[jira] [Created] (HDFS-7725) Incorrect nodes in service metrics caused all writes to fail

2015-02-02 Thread Ming Ma (JIRA)
Ming Ma created HDFS-7725: - Summary: Incorrect nodes in service metrics caused all writes to fail Key: HDFS-7725 URL: https://issues.apache.org/jira/browse/HDFS-7725 Project: Hadoop HDFS Issue

[jira] [Updated] (HDFS-7725) Incorrect nodes in service metrics caused all writes to fail

2015-02-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7725: -- Attachment: HDFS-7725.patch The patch makes sure the nodeInService count won't be updated when a dead node is

[jira] [Updated] (HDFS-7725) Incorrect nodes in service metrics caused all writes to fail

2015-02-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7725: -- Assignee: Ming Ma Status: Patch Available (was: Open) Incorrect nodes in service metrics caused all

[jira] [Updated] (HDFS-7725) Incorrect nodes in service metrics caused all writes to fail

2015-02-02 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-7725: -- Description: One of our clusters sometimes couldn't allocate blocks from any DNs. BlockPlacementPolicyDefault

[jira] [Commented] (HDFS-7314) Aborted DFSClient's impact on long running service like YARN

2015-01-29 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14297133#comment-14297133 ] Ming Ma commented on HDFS-7314: --- Thanks, [~jira.shegalov]. That is interesting. That might

[jira] [Commented] (HDFS-7609) startup used too much time to load edits

2015-01-26 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292690#comment-14292690 ] Ming Ma commented on HDFS-7609: --- [~cnauroth], we upgraded from hadoop 2.0.5 to hadoop 2.4, so

<    1   2   3   4   5   6   7   8   >