[jira] [Commented] (HDFS-14531) Datanode's ScanInfo requires excessive memory

2019-06-07 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16858709#comment-16858709 ] Nathan Roberts commented on HDFS-14531: --- Actually, maybe disabling the DirectoryScanner is more

[jira] [Updated] (HDFS-12441) Suppress UnresolvedPathException in namenode log

2017-09-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-12441: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch

[jira] [Commented] (HDFS-12441) Suppress UnresolvedPathException in namenode log

2017-09-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168433#comment-16168433 ] Nathan Roberts commented on HDFS-12441: --- Cherry picked to branch-3.0, branch-2, and branch-2.8. >

[jira] [Updated] (HDFS-12441) Suppress UnresolvedPathException in namenode log

2017-09-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-12441: -- Fix Version/s: 2.8.3 2.9.0 > Suppress UnresolvedPathException in namenode

[jira] [Updated] (HDFS-12441) Suppress UnresolvedPathException in namenode log

2017-09-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-12441: -- Fix Version/s: 3.0.0-beta1 > Suppress UnresolvedPathException in namenode log >

[jira] [Updated] (HDFS-12441) Suppress UnresolvedPathException in namenode log

2017-09-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-12441: -- Fix Version/s: 3.1.0 > Suppress UnresolvedPathException in namenode log >

[jira] [Commented] (HDFS-12441) Suppress UnresolvedPathException in namenode log

2017-09-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168277#comment-16168277 ] Nathan Roberts commented on HDFS-12441: --- Thanks [~kihwal] for the patch and [~shahrs87] for the

[jira] [Commented] (HDFS-12102) VolumeScanner throttle dropped (fast scan enabled) when there is a corrupt block

2017-08-02 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111696#comment-16111696 ] Nathan Roberts commented on HDFS-12102: --- [~arpitagarwal] Hi Arpit. To provide a bit more background

[jira] [Commented] (HDFS-5042) Completed files lost after power failure

2017-07-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083970#comment-16083970 ] Nathan Roberts commented on HDFS-5042: -- Wondering if we should make this feature configurable. There

[jira] [Commented] (HDFS-12102) VolumeScanner throttle dropped (fast scan enabled) when there is a corrupt block

2017-07-07 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078714#comment-16078714 ] Nathan Roberts commented on HDFS-12102: --- Hi [~aramesh2]. Thanks for the patch. Couple of quick

[jira] [Updated] (HDFS-12102) VolumeScanner throttle dropped (fast scan enabled) when there is a corrupt block

2017-07-07 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-12102: -- Issue Type: New Feature (was: Improvement) > VolumeScanner throttle dropped (fast scan

[jira] [Updated] (HDFS-12102) VolumeScanner throttle dropped (fast scan enabled) when there is a corrupt block

2017-07-07 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-12102: -- Issue Type: Improvement (was: New Feature) > VolumeScanner throttle dropped (fast scan

[jira] [Updated] (HDFS-11818) TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails intermittently

2017-05-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-11818: -- Affects Version/s: 3.0.0-alpha2 Status: Patch Available (was: Open) >

[jira] [Updated] (HDFS-11818) TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails intermittently

2017-05-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-11818: -- Attachment: HDFS-11818.patch HDFS-11818-branch-2.patch Patches for trunk and

[jira] [Commented] (HDFS-11818) TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails intermittently

2017-05-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008325#comment-16008325 ] Nathan Roberts commented on HDFS-11818: --- Know what the issue is. Will post a patch shortly. >

[jira] [Assigned] (HDFS-11818) TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails intermittently

2017-05-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts reassigned HDFS-11818: - Assignee: Nathan Roberts > TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails

[jira] [Commented] (HDFS-11755) Underconstruction blocks can be considered missing

2017-05-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005127#comment-16005127 ] Nathan Roberts commented on HDFS-11755: --- The failing unit tests in trunk have been unstable in

[jira] [Updated] (HDFS-11755) Underconstruction blocks can be considered missing

2017-05-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-11755: -- Attachment: HDFS-11755-branch-2.002.patch HDFS-11755-branch-2.8.002.patch

[jira] [Updated] (HDFS-11755) Underconstruction blocks can be considered missing

2017-05-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-11755: -- Attachment: HDFS-11755.002.patch Fixed Checkstyle Fixed testSetReplicationWhenBatchIBR because

[jira] [Updated] (HDFS-11755) Underconstruction blocks can be considered missing

2017-05-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-11755: -- Status: Patch Available (was: Open) > Underconstruction blocks can be considered missing >

[jira] [Updated] (HDFS-11755) Underconstruction blocks can be considered missing

2017-05-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-11755: -- Status: Open (was: Patch Available) > Underconstruction blocks can be considered missing >

[jira] [Commented] (HDFS-11755) Underconstruction blocks can be considered missing

2017-05-09 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16003459#comment-16003459 ] Nathan Roberts commented on HDFS-11755: --- bq. Do you know which one makes more sense? Not an expert

[jira] [Updated] (HDFS-11755) Underconstruction blocks can be considered missing

2017-05-09 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-11755: -- Status: Patch Available (was: Open) v1 of trunk patch. branch 2 will require a separate

[jira] [Updated] (HDFS-11755) Underconstruction blocks can be considered missing

2017-05-09 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-11755: -- Attachment: HDFS-11755.001.patch > Underconstruction blocks can be considered missing >

[jira] [Created] (HDFS-11755) Underconstruction blocks can be considered missing

2017-05-04 Thread Nathan Roberts (JIRA)
Nathan Roberts created HDFS-11755: - Summary: Underconstruction blocks can be considered missing Key: HDFS-11755 URL: https://issues.apache.org/jira/browse/HDFS-11755 Project: Hadoop HDFS

[jira] [Commented] (HDFS-11752) getNonDfsUsed return 0 if reserved bigger than actualNonDfsUsed

2017-05-04 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996815#comment-15996815 ] Nathan Roberts commented on HDFS-11752: --- I think this is intentional. My understanding is that

[jira] [Comment Edited] (HDFS-11661) GetContentSummary uses excessive amounts of memory

2017-04-20 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977163#comment-15977163 ] Nathan Roberts edited comment on HDFS-11661 at 4/20/17 6:05 PM:

[jira] [Commented] (HDFS-11661) GetContentSummary uses excessive amounts of memory

2017-04-20 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977163#comment-15977163 ] Nathan Roberts commented on HDFS-11661: --- [~jojochuang], sure, occasional `du -s` of very large

[jira] [Commented] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice

2017-04-18 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973105#comment-15973105 ] Nathan Roberts commented on HDFS-10797: --- After deploying this to a cluster with a few hundred nodes,

[jira] [Created] (HDFS-11661) GetContentSummary uses excessive amounts of memory

2017-04-17 Thread Nathan Roberts (JIRA)
Nathan Roberts created HDFS-11661: - Summary: GetContentSummary uses excessive amounts of memory Key: HDFS-11661 URL: https://issues.apache.org/jira/browse/HDFS-11661 Project: Hadoop HDFS

[jira] [Updated] (HDFS-4660) Block corruption can happen during pipeline recovery

2016-08-25 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-4660: - Attachment: periodic_hflush.patch > Block corruption can happen during pipeline recovery >

[jira] [Commented] (HDFS-4660) Block corruption can happen during pipeline recovery

2016-08-25 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437036#comment-15437036 ] Nathan Roberts commented on HDFS-4660: -- Hi [~yzhangal]. Had to go back to an old git stash, but I'll

[jira] [Commented] (HDFS-9239) DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness

2016-03-30 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218470#comment-15218470 ] Nathan Roberts commented on HDFS-9239: -- bq. Just to make sure I'm clear, are you talking about

[jira] [Commented] (HDFS-9239) DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness

2016-03-30 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218302#comment-15218302 ] Nathan Roberts commented on HDFS-9239: -- bq. However,making it lighter on the datanode side is a good

[jira] [Updated] (HDFS-4946) Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable

2016-02-18 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-4946: - Affects Version/s: 3.0.0 2.7.2 Target Version/s: 3.0.0, 2.8.0

[jira] [Reopened] (HDFS-4946) Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable

2016-02-18 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts reopened HDFS-4946: -- [~jrkinley], re-opening because this is a very useful patch. Let me know if you disagree or would

[jira] [Updated] (HDFS-4946) Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable

2016-02-18 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-4946: - Attachment: HDFS-4946-2.patch Uploaded a new version of this patch for trunk. We have found this

[jira] [Commented] (HDFS-7060) Avoid taking locks when sending heartbeats from the DataNode

2015-11-20 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15018828#comment-15018828 ] Nathan Roberts commented on HDFS-7060: -- Has anyone given any further thought on this patch? It seems

[jira] [Commented] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4

2015-11-19 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014463#comment-15014463 ] Nathan Roberts commented on HDFS-8791: -- Thanks [~ctrezzo] for the patch! Nice writeup on the

[jira] [Commented] (HDFS-8873) throttle directoryScanner

2015-09-24 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906603#comment-14906603 ] Nathan Roberts commented on HDFS-8873: -- Thanks [~templedf]. I like that the stopwatch class makes this

[jira] [Commented] (HDFS-8873) throttle directoryScanner

2015-09-24 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906919#comment-14906919 ] Nathan Roberts commented on HDFS-8873: -- Thanks [~templedf] for the update! I'm +1 (non-binding) for v9

[jira] [Commented] (HDFS-8873) throttle directoryScanner

2015-09-23 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904605#comment-14904605 ] Nathan Roberts commented on HDFS-8873: -- Thanks [~templedf] for the update. I am sorry I haven't had a

[jira] [Commented] (HDFS-8873) throttle directoryScanner

2015-09-18 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876736#comment-14876736 ] Nathan Roberts commented on HDFS-8873: -- bq. Sure. It won't matter since stop() is only called by

[jira] [Commented] (HDFS-8873) throttle directoryScanner

2015-09-18 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876500#comment-14876500 ] Nathan Roberts commented on HDFS-8873: -- Thanks [~templedf] for the patch. A few comments.

[jira] [Created] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets

2015-08-13 Thread Nathan Roberts (JIRA)
Nathan Roberts created HDFS-8894: Summary: Set SO_KEEPALIVE on DN server sockets Key: HDFS-8894 URL: https://issues.apache.org/jira/browse/HDFS-8894 Project: Hadoop HDFS Issue Type: Bug

[jira] [Created] (HDFS-8873) throttle directoryScanner

2015-08-07 Thread Nathan Roberts (JIRA)
Nathan Roberts created HDFS-8873: Summary: throttle directoryScanner Key: HDFS-8873 URL: https://issues.apache.org/jira/browse/HDFS-8873 Project: Hadoop HDFS Issue Type: Bug

[jira] [Commented] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4

2015-08-04 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654426#comment-14654426 ] Nathan Roberts commented on HDFS-8791: -- My preference would be to take a smaller

[jira] [Commented] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4

2015-08-04 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14653974#comment-14653974 ] Nathan Roberts commented on HDFS-8791: -- Curious what folks would think about going

[jira] [Commented] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab

2015-07-21 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14635299#comment-14635299 ] Nathan Roberts commented on HDFS-6407: -- My understanding is that the Legacy UI was

[jira] [Commented] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4

2015-07-21 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14635908#comment-14635908 ] Nathan Roberts commented on HDFS-8791: -- bq. I'm having trouble understanding these

[jira] [Commented] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4

2015-07-20 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633669#comment-14633669 ] Nathan Roberts commented on HDFS-8791: -- Hi [~cmccabe]. Thanks for the idea. Yes, I had

[jira] [Commented] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4

2015-07-20 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633804#comment-14633804 ] Nathan Roberts commented on HDFS-8791: -- I agree we should optimize all the potential

[jira] [Commented] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4

2015-07-20 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633675#comment-14633675 ] Nathan Roberts commented on HDFS-8791: -- I forgot to mention that I'm pretty confident

[jira] [Commented] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4

2015-07-17 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631334#comment-14631334 ] Nathan Roberts commented on HDFS-8791: -- Sure. Randomly sampled node had 4 4TB drives,

[jira] [Created] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4

2015-07-16 Thread Nathan Roberts (JIRA)
Nathan Roberts created HDFS-8791: Summary: block ID-based DN storage layout can be very slow for datanode on ext4 Key: HDFS-8791 URL: https://issues.apache.org/jira/browse/HDFS-8791 Project: Hadoop

[jira] [Commented] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4

2015-07-16 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630493#comment-14630493 ] Nathan Roberts commented on HDFS-8791: -- For reference, the stack trace when du is

[jira] [Commented] (HDFS-4660) Block corruption can happen during pipeline recovery

2015-06-16 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14588695#comment-14588695 ] Nathan Roberts commented on HDFS-4660: -- +1 on the patch. I have reviewed the patch

[jira] [Updated] (HDFS-8404) pending block replication can get stuck using older genstamp

2015-05-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-8404: - Attachment: HDFS-8404-v1.patch Thanks [~kihwal]. Updated based on latest trunk. pending block

[jira] [Updated] (HDFS-8404) pending block replication can get stuck using older genstamp

2015-05-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-8404: - Status: Open (was: Patch Available) pending block replication can get stuck using older genstamp

[jira] [Updated] (HDFS-8404) pending block replication can get stuck using older genstamp

2015-05-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-8404: - Status: Patch Available (was: Open) pending block replication can get stuck using older genstamp

[jira] [Created] (HDFS-8404) pending block replication can get stuck using older genstamp

2015-05-14 Thread Nathan Roberts (JIRA)
Nathan Roberts created HDFS-8404: Summary: pending block replication can get stuck using older genstamp Key: HDFS-8404 URL: https://issues.apache.org/jira/browse/HDFS-8404 Project: Hadoop HDFS

[jira] [Updated] (HDFS-8404) pending block replication can get stuck using older genstamp

2015-05-14 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-8404: - Attachment: HDFS-8404-v0.patch pending block replication can get stuck using older genstamp

[jira] [Updated] (HDFS-8404) pending block replication can get stuck using older genstamp

2015-05-14 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-8404: - Status: Patch Available (was: Open) pending block replication can get stuck using older genstamp

[jira] [Commented] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized

2015-04-02 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14393437#comment-14393437 ] Nathan Roberts commented on HDFS-8041: -- Hi [~kihwal]. Some minor comments on the patch

[jira] [Commented] (HDFS-7742) favoring decommissioning node for replication can cause a block to stay underreplicated for long periods

2015-03-30 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386724#comment-14386724 ] Nathan Roberts commented on HDFS-7742: -- Thanks for the review nicholas! There is a

[jira] [Updated] (HDFS-7742) favoring decommissioning node for replication can cause a block to stay underreplicated for long periods

2015-03-27 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-7742: - Target Version/s: 3.0.0 (was: 3.0.0, 2.7.0) favoring decommissioning node for replication can

[jira] [Updated] (HDFS-7742) favoring decommissioning node for replication can cause a block to stay underreplicated for long periods

2015-03-27 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-7742: - Attachment: HDFS-7742-v0.patch Attached patch. Favors decommissioning nodes a bit by allowing them

[jira] [Updated] (HDFS-7742) favoring decommissioning node for replication can cause a block to stay underreplicated for long periods

2015-03-27 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-7742: - Target Version/s: 3.0.0, 2.7.0 (was: 3.0.0) Status: Patch Available (was: Open)

[jira] [Created] (HDFS-7742) favoring decommissioning node for replication can cause a block to stay underreplicated for long periods

2015-02-06 Thread Nathan Roberts (JIRA)
Nathan Roberts created HDFS-7742: Summary: favoring decommissioning node for replication can cause a block to stay underreplicated for long periods Key: HDFS-7742 URL:

[jira] [Created] (HDFS-7645) Rolling upgrade is restoring blocks from trash multiple times

2015-01-20 Thread Nathan Roberts (JIRA)
Nathan Roberts created HDFS-7645: Summary: Rolling upgrade is restoring blocks from trash multiple times Key: HDFS-7645 URL: https://issues.apache.org/jira/browse/HDFS-7645 Project: Hadoop HDFS

[jira] [Commented] (HDFS-7548) Corrupt block reporting delayed until datablock scanner thread detects it

2015-01-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273764#comment-14273764 ] Nathan Roberts commented on HDFS-7548: -- - I think we need to prioritize a scan for

[jira] [Commented] (HDFS-7548) Corrupt block reporting delayed until datablock scanner thread detects it

2015-01-07 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268347#comment-14268347 ] Nathan Roberts commented on HDFS-7548: -- I think we need to handle the

[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN

2014-12-16 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248733#comment-14248733 ] Nathan Roberts commented on HDFS-2825: -- [~tlipcon] - It might help. I'm a little

[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN

2014-12-02 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14232257#comment-14232257 ] Nathan Roberts commented on HDFS-2825: -- Any chance folks have altered their view on

[jira] [Commented] (HDFS-6658) Namenode memory optimization - Block replicas list

2014-07-16 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064095#comment-14064095 ] Nathan Roberts commented on HDFS-6658: -- {quote} I guess my argument is that (in the

[jira] [Commented] (HDFS-6658) Namenode memory optimization - Block replicas list

2014-07-14 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060740#comment-14060740 ] Nathan Roberts commented on HDFS-6658: -- Maybe have a simple fragmentation metric and

[jira] [Commented] (HDFS-6584) Support archival storage

2014-07-14 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14060815#comment-14060815 ] Nathan Roberts commented on HDFS-6584: -- As part of this work, would it make sense to

[jira] [Commented] (HDFS-6658) Namenode memory optimization - Block replicas list

2014-07-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057858#comment-14057858 ] Nathan Roberts commented on HDFS-6658: -- Thanks Amir. Very attractive memory savings.

[jira] [Created] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab

2014-05-16 Thread Nathan Roberts (JIRA)
Nathan Roberts created HDFS-6407: Summary: new namenode UI, lost ability to sort columns in datanode tab Key: HDFS-6407 URL: https://issues.apache.org/jira/browse/HDFS-6407 Project: Hadoop HDFS

[jira] [Commented] (HDFS-6247) Avoid timeouts for replaceBlock() call by sending intermediate responses to Balancer

2014-04-16 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13971837#comment-13971837 ] Nathan Roberts commented on HDFS-6247: -- +1 This will be better than the longish

[jira] [Updated] (HDFS-6166) revisit balancer so_timeout

2014-03-31 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-6166: - Attachment: HDFS-6166-branch23.patch Patch for branch 23 revisit balancer so_timeout

[jira] [Commented] (HDFS-6166) revisit balancer so_timeout

2014-03-28 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951461#comment-13951461 ] Nathan Roberts commented on HDFS-6166: -- Tested on a 400 node cluster with a bandwidth

[jira] [Commented] (HDFS-6166) revisit balancer so_timeout

2014-03-28 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951493#comment-13951493 ] Nathan Roberts commented on HDFS-6166: -- Maybe our two comments passed in the mail. Yes

[jira] [Commented] (HDFS-6166) revisit balancer so_timeout

2014-03-28 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13951509#comment-13951509 ] Nathan Roberts commented on HDFS-6166: -- The blocks don't have to be very large. There

[jira] [Created] (HDFS-6166) revisit balancer so_timeout

2014-03-27 Thread Nathan Roberts (JIRA)
Nathan Roberts created HDFS-6166: Summary: revisit balancer so_timeout Key: HDFS-6166 URL: https://issues.apache.org/jira/browse/HDFS-6166 Project: Hadoop HDFS Issue Type: Bug

[jira] [Updated] (HDFS-6166) revisit balancer so_timeout

2014-03-27 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-6166: - Attachment: HDFS-6166.patch proposed patch. revisit balancer so_timeout

[jira] [Updated] (HDFS-6166) revisit balancer so_timeout

2014-03-27 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-6166: - Target Version/s: 3.0.0, 2.4.0 (was: 2.4.0) Status: Patch Available (was: Open)

[jira] [Updated] (HDFS-5806) balancer should set SoTimeout to avoid indefinite hangs

2014-02-26 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-5806: - Attachment: HDFS-5806-0.23.patch 0.23 version of patch balancer should set SoTimeout to avoid

[jira] [Updated] (HDFS-5788) listLocatedStatus response can be very large

2014-01-22 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-5788: - Attachment: HDFS-5788.patch patch for trunk. listLocatedStatus response can be very large

[jira] [Updated] (HDFS-5788) listLocatedStatus response can be very large

2014-01-22 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-5788: - Status: Patch Available (was: Open) listLocatedStatus response can be very large

[jira] [Commented] (HDFS-5806) balancer should set SoTimeout to avoid indefinite hangs

2014-01-22 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13879273#comment-13879273 ] Nathan Roberts commented on HDFS-5806: -- Andrew, thanks for taking a look. Sorry about

[jira] [Created] (HDFS-5806) balancer should set SoTimeout to avoid indefinite hangs

2014-01-21 Thread Nathan Roberts (JIRA)
Nathan Roberts created HDFS-5806: Summary: balancer should set SoTimeout to avoid indefinite hangs Key: HDFS-5806 URL: https://issues.apache.org/jira/browse/HDFS-5806 Project: Hadoop HDFS

[jira] [Updated] (HDFS-5806) balancer should set SoTimeout to avoid indefinite hangs

2014-01-21 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-5806: - Attachment: HDFS-5806.patch use setSoTimeout() to avoid read hangs. balancer should set

[jira] [Updated] (HDFS-5806) balancer should set SoTimeout to avoid indefinite hangs

2014-01-21 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-5806: - Status: Patch Available (was: Open) balancer should set SoTimeout to avoid indefinite hangs

[jira] [Created] (HDFS-5788) listLocatedStatus response can be very large

2014-01-16 Thread Nathan Roberts (JIRA)
Nathan Roberts created HDFS-5788: Summary: listLocatedStatus response can be very large Key: HDFS-5788 URL: https://issues.apache.org/jira/browse/HDFS-5788 Project: Hadoop HDFS Issue Type:

[jira] [Commented] (HDFS-5788) listLocatedStatus response can be very large

2014-01-16 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13874107#comment-13874107 ] Nathan Roberts commented on HDFS-5788: -- A simple solution is: Restrict the size to

[jira] [Updated] (HDFS-5477) Block manager as a service

2013-12-17 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-5477: - Attachment: Standalone BM.pdf Re-attach standalone pdf to fix graphics. Block manager as a

[jira] [Updated] (HDFS-5477) Block manager as a service

2013-12-11 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-5477: - Attachment: Proposal.pdf Fix formatting problems in PDF. Block manager as a service

[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades

2013-12-02 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13836540#comment-13836540 ] Nathan Roberts commented on HDFS-5535: -- Hi. Initial draft is still a little rough.

[jira] [Updated] (HDFS-5496) Make replication queue initialization asynchronous

2013-11-21 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated HDFS-5496: - Issue Type: Sub-task (was: Improvement) Parent: HDFS-5535 Make replication queue

  1   2   >