[jira] [Commented] (YARN-7333) container-executor fails to remove entries from a directory that is not writable or executable

2017-10-16 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206580#comment-16206580 ] Nathan Roberts commented on YARN-7333: -- Thanks Jason. I'll commit this shortly. > container-executor

[jira] [Commented] (YARN-7333) container-executor fails to remove entries from a directory that is not writable or executable

2017-10-16 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206438#comment-16206438 ] Nathan Roberts commented on YARN-7333: -- Thanks for the patch Jason. A couple of comments: - Log line

[jira] [Updated] (YARN-6219) NM web server related UT fails with "NMWebapps failed to start."

2017-09-08 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-6219: - Fix Version/s: 2.9.0 > NM web server related UT fails with "NMWebapps failed to start." >

[jira] [Commented] (YARN-6219) NM web server related UT fails with "NMWebapps failed to start."

2017-09-08 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159308#comment-16159308 ] Nathan Roberts commented on YARN-6219: -- +1. Thanks [~jlowe]. I will commit this shortly. > NM web

[jira] [Commented] (YARN-6763) TestProcfsBasedProcessTree#testProcessTree fails in trunk

2017-08-18 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133624#comment-16133624 ] Nathan Roberts commented on YARN-6763: -- Sorry it took so long to get back to this issue. My feeling

[jira] [Updated] (YARN-7014) container-executor has off-by-one error which can corrupt the heap

2017-08-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-7014: - Fix Version/s: 3.0.0-beta1 > container-executor has off-by-one error which can corrupt the heap >

[jira] [Commented] (YARN-7014) container-executor has off-by-one error which can corrupt the heap

2017-08-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127824#comment-16127824 ] Nathan Roberts commented on YARN-7014: -- +1 on the patch. I will commit shortly. Thanks [~jlowe] for

[jira] [Assigned] (YARN-6867) AbstractYarnScheduler reports the configured maximum resources, instead of the actual, even after the configured waittime

2017-07-25 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts reassigned YARN-6867: Assignee: Nathan Roberts > AbstractYarnScheduler reports the configured maximum resources,

[jira] [Commented] (YARN-6775) CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls

2017-07-17 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089936#comment-16089936 ] Nathan Roberts commented on YARN-6775: -- Attached screenshots that show a couple of before/after

[jira] [Updated] (YARN-6775) CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls

2017-07-17 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-6775: - Attachment: rpcprocessingtimeschedulerport.png > CapacityScheduler: Improvements to

[jira] [Updated] (YARN-6775) CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls

2017-07-17 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-6775: - Attachment: rmeventprocbusy.png > CapacityScheduler: Improvements to assignContainers, avoid

[jira] [Commented] (YARN-6775) CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls

2017-07-17 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089894#comment-16089894 ] Nathan Roberts commented on YARN-6775: -- [~leftnoteasy], I applied YARN-6775.branch-2.002.patch to

[jira] [Updated] (YARN-6775) CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls

2017-07-14 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-6775: - Attachment: YARN-6775.branch-2.8.002.patch > CapacityScheduler: Improvements to assignContainers,

[jira] [Commented] (YARN-6768) Improve performance of yarn api record toString and fromString

2017-07-13 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086522#comment-16086522 ] Nathan Roberts commented on YARN-6768: -- Probably don't need to calculate full numDigits. once you have

[jira] [Updated] (YARN-6775) CapacityScheduler: Improvements to assignContainers()

2017-07-13 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-6775: - Attachment: YARN-6775.branch-2.002.patch > CapacityScheduler: Improvements to assignContainers() >

[jira] [Commented] (YARN-6775) CapacityScheduler: Improvements to assignContainers()

2017-07-13 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085715#comment-16085715 ] Nathan Roberts commented on YARN-6775: -- Thanks [~leftnoteasy]. I looked at UT failures. I've seen

[jira] [Updated] (YARN-6775) CapacityScheduler: Improvements to assignContainers()

2017-07-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-6775: - Attachment: YARN-6775.002.patch Addressed checkstyle warnings and renamed rsrvd as requested >

[jira] [Commented] (YARN-6775) CapacityScheduler: Improvements to assignContainers()

2017-07-11 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083045#comment-16083045 ] Nathan Roberts commented on YARN-6775: -- Thanks [~leftnoteasy] for the review. bq. 1)

[jira] [Commented] (YARN-6797) TimelineWriter does not fully consume the POST response

2017-07-11 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082242#comment-16082242 ] Nathan Roberts commented on YARN-6797: -- Thanks Jason. +1 on this patch. > TimelineWriter does not

[jira] [Commented] (YARN-6763) TestProcfsBasedProcessTree#testProcessTree fails in trunk

2017-07-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080345#comment-16080345 ] Nathan Roberts commented on YARN-6763: -- [~bibinchundatt] - Out of curiosity, what is the OS

[jira] [Commented] (YARN-6775) CapacityScheduler: Improvements to assignContainers()

2017-07-07 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078677#comment-16078677 ] Nathan Roberts commented on YARN-6775: -- Below is the list of changes included in the patch. Each is

[jira] [Updated] (YARN-6775) CapacityScheduler: Improvements to assignContainers()

2017-07-07 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-6775: - Attachment: YARN-6775.001.patch > CapacityScheduler: Improvements to assignContainers() >

[jira] [Created] (YARN-6775) CapacityScheduler: Improvements to assignContainers()

2017-07-07 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-6775: Summary: CapacityScheduler: Improvements to assignContainers() Key: YARN-6775 URL: https://issues.apache.org/jira/browse/YARN-6775 Project: Hadoop YARN

[jira] [Commented] (YARN-6768) Improve performance of yarn api record toString and fromString

2017-07-07 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078331#comment-16078331 ] Nathan Roberts commented on YARN-6768: -- Thanks Jon! As a datapoint, I have a testcase that measures

[jira] [Commented] (YARN-6763) TestProcfsBasedProcessTree#testProcessTree fails in trunk

2017-07-07 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078091#comment-16078091 ] Nathan Roberts commented on YARN-6763: -- [~bibinchundatt] thanks for reporting this. I'll take a look

[jira] [Assigned] (YARN-6763) TestProcfsBasedProcessTree#testProcessTree fails in trunk

2017-07-07 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts reassigned YARN-6763: Assignee: Nathan Roberts > TestProcfsBasedProcessTree#testProcessTree fails in trunk >

[jira] [Updated] (YARN-6649) RollingLevelDBTimelineServer throws RuntimeException if object decoding ever fails runtime exception

2017-05-31 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-6649: - Fix Version/s: 2.8.2 2.8.1 2.9.0 >

[jira] [Updated] (YARN-6649) RollingLevelDBTimelineServer throws RuntimeException if object decoding ever fails runtime exception

2017-05-31 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-6649: - Fix Version/s: 3.0.0-alpha4 > RollingLevelDBTimelineServer throws RuntimeException if object

[jira] [Commented] (YARN-6649) RollingLevelDBTimelineServer throws RuntimeException if object decoding ever fails runtime exception

2017-05-30 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030211#comment-16030211 ] Nathan Roberts commented on YARN-6649: -- Thanks Jon for the explanation. I'll commit this tomorrow

[jira] [Commented] (YARN-6649) RollingLevelDBTimelineServer throws RuntimeException if object decoding ever fails runtime exception

2017-05-30 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030070#comment-16030070 ] Nathan Roberts commented on YARN-6649: -- Thanks Jon for the patch. Since there is no unit test, could

[jira] [Comment Edited] (YARN-6585) RM fails to start when upgrading from 2.7 to 2.8 for clusters with node labels.

2017-05-11 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007016#comment-16007016 ] Nathan Roberts edited comment on YARN-6585 at 5/11/17 7:13 PM: --- YARN-6143

[jira] [Commented] (YARN-6585) RM fails to start when upgrading from 2.7 to 2.8 for clusters with node labels.

2017-05-11 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007016#comment-16007016 ] Nathan Roberts commented on YARN-6585: -- YARN-6143 changed AddToClusterNodeLabelsRequestProto such that

[jira] [Commented] (YARN-6344) Rethinking OFF_SWITCH locality in CapacityScheduler

2017-03-20 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932783#comment-15932783 ] Nathan Roberts commented on YARN-6344: -- +1 on improving localityWaitFactor. It definitely won't behave

[jira] [Commented] (YARN-5179) Issue of CPU usage of containers

2017-02-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854387#comment-15854387 ] Nathan Roberts commented on YARN-5179: -- bq. I think ResourceUtilization.getCPU() has a similar sort of

[jira] [Commented] (YARN-5179) Issue of CPU usage of containers

2017-02-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854374#comment-15854374 ] Nathan Roberts commented on YARN-5179: -- I think ResourceUtilization.getCPU() has a similar sort of

[jira] [Commented] (YARN-2904) Use linux cgroups to enhance container tear down

2016-12-16 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15754975#comment-15754975 ] Nathan Roberts commented on YARN-2904: -- Simple streaming job that does the following illustrates tasks

[jira] [Commented] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

2016-10-31 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622245#comment-15622245 ] Nathan Roberts commented on YARN-5356: -- Thanks [~elgoiri] for the update. +1 (non-binding) on version

[jira] [Commented] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

2016-10-28 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15616725#comment-15616725 ] Nathan Roberts commented on YARN-5356: -- Hi [~elgoiri]. Looked over version 6 of the patch. I am

[jira] [Updated] (YARN-4963) capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable

2016-10-27 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4963: - Attachment: YARN-4963.004.patch rebased with trunk > capacity scheduler: Make number of

[jira] [Commented] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

2016-09-19 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504823#comment-15504823 ] Nathan Roberts commented on YARN-5356: -- Hi [~elgoiri]. Tried out the patch but get NPE in RM because

[jira] [Commented] (YARN-3432) Cluster metrics have wrong Total Memory when there is reserved memory on CS

2016-09-01 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456537#comment-15456537 ] Nathan Roberts commented on YARN-3432: -- Recently ran into this issue again. Just seems wrong that

[jira] [Commented] (YARN-5202) Dynamic Overcommit of Node Resources - POC

2016-08-26 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440131#comment-15440131 ] Nathan Roberts commented on YARN-5202: -- Yahoo has received a letter accusing the patches originally

[jira] [Updated] (YARN-5202) Dynamic Overcommit of Node Resources - POC

2016-08-26 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-5202: - Attachment: (was: YARN-5202.patch) > Dynamic Overcommit of Node Resources - POC >

[jira] [Updated] (YARN-5202) Dynamic Overcommit of Node Resources - POC

2016-08-26 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-5202: - Attachment: (was: YARN-5202-branch2.7-uber.patch) > Dynamic Overcommit of Node Resources - POC

[jira] [Commented] (YARN-5551) Ignore deleted file mapping from memory computation when smaps is enabled

2016-08-25 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437415#comment-15437415 ] Nathan Roberts commented on YARN-5551: -- I think the two examples you provided in the description are

[jira] [Commented] (YARN-5551) Ignore deleted file mapping from memory computation when smaps is enabled

2016-08-24 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435726#comment-15435726 ] Nathan Roberts commented on YARN-5551: -- bq. Nathan Roberts, Jason Lowe - do you mind reviewing the

[jira] [Commented] (YARN-5352) Allow container-executor to use private /tmp

2016-08-24 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435125#comment-15435125 ] Nathan Roberts commented on YARN-5352: -- bq. I think minimally this needs to be an optional feature

[jira] [Created] (YARN-5540) Capacity Scheduler spends too much time looking at empty priorities

2016-08-19 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-5540: Summary: Capacity Scheduler spends too much time looking at empty priorities Key: YARN-5540 URL: https://issues.apache.org/jira/browse/YARN-5540 Project: Hadoop YARN

[jira] [Updated] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit

2016-08-18 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-3388: - Attachment: YARN-3388-v7.patch Thanks [~leftnoteasy] for the comments. I took both suggestions in

[jira] [Updated] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit

2016-08-17 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-3388: - Attachment: YARN-3388-v6.patch Cleaned up most of the findbugs/checkstyle issues. > Allocation in

[jira] [Updated] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit

2016-08-16 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-3388: - Attachment: YARN-3388-v5.patch fixed build error. > Allocation in LeafQueue could get stuck

[jira] [Updated] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit

2016-08-16 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-3388: - Attachment: YARN-3388-v4.patch Thanks for the comments [~jlowe]. upmerged and added whitespace. >

[jira] [Commented] (YARN-5352) Allow container-executor to use private /tmp

2016-07-19 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384496#comment-15384496 ] Nathan Roberts commented on YARN-5352: -- This patch doesn't address localization. Thinking was that

[jira] [Updated] (YARN-5352) Allow container-executor to use private /tmp

2016-07-19 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-5352: - Attachment: YARN-5352-v0.patch Patch that uses linux private namespace and bind mounts to achieve

[jira] [Commented] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

2016-07-18 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383184#comment-15383184 ] Nathan Roberts commented on YARN-5356: -- Hi Inigo, couple more comments when looking over most recent

[jira] [Updated] (YARN-5202) Dynamic Overcommit of Node Resources - POC

2016-07-18 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-5202: - Attachment: YARN-5202-branch2.7-uber.patch Attached an uber patch for branch 2.7 so that folks can

[jira] [Commented] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

2016-07-18 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382613#comment-15382613 ] Nathan Roberts commented on YARN-5356: -- Thanks [~elgoiri] for the patch! Some quick comments

[jira] [Updated] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

2016-07-13 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-5356: - Description: Currently ResourceUtilization contains absolute quantities of resource used (e.g.

[jira] [Updated] (YARN-5356) NodeManager should communicate physical resource capability to ResourceManager

2016-07-13 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-5356: - Summary: NodeManager should communicate physical resource capability to ResourceManager (was:

[jira] [Commented] (YARN-5356) ResourceUtilization should also include resource availability

2016-07-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373394#comment-15373394 ] Nathan Roberts commented on YARN-5356: -- bq. I can post a patch with these changes if you want. That

[jira] [Comment Edited] (YARN-5356) ResourceUtilization should also include resource availability

2016-07-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372905#comment-15372905 ] Nathan Roberts edited comment on YARN-5356 at 7/12/16 2:07 PM: --- bq. Nathan

[jira] [Commented] (YARN-5356) ResourceUtilization should also include resource availability

2016-07-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372905#comment-15372905 ] Nathan Roberts commented on YARN-5356: -- bq. Nathan Roberts, I understand that your problem is that

[jira] [Created] (YARN-5356) ResourceUtilization should also include resource availability

2016-07-11 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-5356: Summary: ResourceUtilization should also include resource availability Key: YARN-5356 URL: https://issues.apache.org/jira/browse/YARN-5356 Project: Hadoop YARN

[jira] [Created] (YARN-5352) Allow container-executor to use private /tmp

2016-07-11 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-5352: Summary: Allow container-executor to use private /tmp Key: YARN-5352 URL: https://issues.apache.org/jira/browse/YARN-5352 Project: Hadoop YARN Issue Type:

[jira] [Commented] (YARN-5215) Scheduling containers based on external load in the servers

2016-06-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332601#comment-15332601 ] Nathan Roberts commented on YARN-5215: -- Thanks [~elgoiri] for the work. Maybe Summit would be a good

[jira] [Updated] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit

2016-06-14 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-3388: - Attachment: YARN-3388-v3.patch [~leftnoteasy], [~eepayne]. Ok, "soon" was extremely relative;)

[jira] [Commented] (YARN-5214) Pending on synchronized method DirectoryCollection#checkDirs can hang NM's NodeStatusUpdater

2016-06-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324682#comment-15324682 ] Nathan Roberts commented on YARN-5214: -- [~djp]. I agree it makes sense to keep the heartbeat path as

[jira] [Commented] (YARN-5214) Pending on synchronized method DirectoryCollection#checkDirs can hang NM's NodeStatusUpdater

2016-06-08 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321342#comment-15321342 ] Nathan Roberts commented on YARN-5214: -- I'm not suggesting this change shouldn't be made but keep in

[jira] [Updated] (YARN-5202) Dynamic Overcommit of Node Resources - POC

2016-06-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-5202: - Attachment: YARN-5202.patch Originally branched from commit:

[jira] [Created] (YARN-5202) Dynamic Overcommit of Node Resources - POC

2016-06-06 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-5202: Summary: Dynamic Overcommit of Node Resources - POC Key: YARN-5202 URL: https://issues.apache.org/jira/browse/YARN-5202 Project: Hadoop YARN Issue Type:

[jira] [Updated] (YARN-4963) capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable

2016-05-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4963: - Attachment: YARN-4963.003.patch Thank you [~leftnoteasy] for the comments! I have addressed them

[jira] [Commented] (YARN-5039) Applications ACCEPTED but not starting

2016-05-11 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280191#comment-15280191 ] Nathan Roberts commented on YARN-5039: -- Thanks [~milesc]. This seems to be an Amazon emr thing (unless

[jira] [Commented] (YARN-5039) Applications ACCEPTED but not starting

2016-05-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279018#comment-15279018 ] Nathan Roberts commented on YARN-5039: -- Thanks [~milesc]! Still not quite enough. How about

[jira] [Commented] (YARN-4963) capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable

2016-05-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278369#comment-15278369 ] Nathan Roberts commented on YARN-4963: -- I don't believe test failures are related to this change. If

[jira] [Commented] (YARN-5039) Applications ACCEPTED but not starting

2016-05-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278314#comment-15278314 ] Nathan Roberts commented on YARN-5039: -- [~milesc], if you have it in this state again, would you mind

[jira] [Updated] (YARN-4963) capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable

2016-05-05 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4963: - Attachment: YARN-4963.002.patch Address checkstyle comment. > capacity scheduler: Make number

[jira] [Commented] (YARN-4963) capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable

2016-04-28 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263109#comment-15263109 ] Nathan Roberts commented on YARN-4963: -- Sorry it took so long to get back to this. I filed YARN-5013

[jira] [Commented] (YARN-5013) Allow applications to provide input on amount of locality delay to use

2016-04-28 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263106#comment-15263106 ] Nathan Roberts commented on YARN-5013: -- Re-posting latest comment from [~Naganarasimha] Thanks for

[jira] [Created] (YARN-5013) Allow applications to provide input on amount of locality delay to use

2016-04-28 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-5013: Summary: Allow applications to provide input on amount of locality delay to use Key: YARN-5013 URL: https://issues.apache.org/jira/browse/YARN-5013 Project: Hadoop

[jira] [Commented] (YARN-5008) LeveldbRMStateStore database can grow substantially leading to long recovery times

2016-04-28 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262472#comment-15262472 ] Nathan Roberts commented on YARN-5008: -- Thanks for the patch. LGTM. +1 non-binding >

[jira] [Updated] (YARN-5003) Add container resource to RM audit log

2016-04-27 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-5003: - Attachment: YARN-5003.001.patch Attaching patch > Add container resource to RM audit log >

[jira] [Created] (YARN-5003) Add container resource to RM audit log

2016-04-27 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-5003: Summary: Add container resource to RM audit log Key: YARN-5003 URL: https://issues.apache.org/jira/browse/YARN-5003 Project: Hadoop YARN Issue Type:

[jira] [Commented] (YARN-4556) TestFifoScheduler.testResourceOverCommit fails

2016-04-21 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252593#comment-15252593 ] Nathan Roberts commented on YARN-4556: -- Patch seems like a reasonable test improvement. +1 non-binding

[jira] [Commented] (YARN-4963) capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable

2016-04-20 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250137#comment-15250137 ] Nathan Roberts commented on YARN-4963: -- bq. IMO, I think application specific configurations should be

[jira] [Commented] (YARN-4963) capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable

2016-04-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243662#comment-15243662 ] Nathan Roberts commented on YARN-4963: -- Thanks [~leftnoteasy] for the feedback. I agree that it would

[jira] [Updated] (YARN-4964) Allow ShuffleHandler readahead without drop-behind

2016-04-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4964: - Attachment: YARN-4964.001.patch > Allow ShuffleHandler readahead without drop-behind >

[jira] [Created] (YARN-4964) Allow ShuffleHandler readahead without drop-behind

2016-04-15 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-4964: Summary: Allow ShuffleHandler readahead without drop-behind Key: YARN-4964 URL: https://issues.apache.org/jira/browse/YARN-4964 Project: Hadoop YARN Issue

[jira] [Updated] (YARN-4963) capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable

2016-04-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4963: - Attachment: YARN-4963.001.patch > capacity scheduler: Make number of OFF_SWITCH assignments per

[jira] [Created] (YARN-4963) capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable

2016-04-15 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-4963: Summary: capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable Key: YARN-4963 URL: https://issues.apache.org/jira/browse/YARN-4963

[jira] [Commented] (YARN-4924) NM recovery race can lead to container not cleaned up

2016-04-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15228258#comment-15228258 ] Nathan Roberts commented on YARN-4924: -- Sorry [~sandflee]. I missed your comment about updating

[jira] [Updated] (YARN-4924) NM recovery race can lead to container not cleaned up

2016-04-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4924: - Assignee: (was: Nathan Roberts) > NM recovery race can lead to container not cleaned up >

[jira] [Commented] (YARN-4924) NM recovery race can lead to container not cleaned up

2016-04-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15228243#comment-15228243 ] Nathan Roberts commented on YARN-4924: -- Thanks [~sandflee], [~jlowe] for the suggestion. I'll work up

[jira] [Assigned] (YARN-4924) NM recovery race can lead to container not cleaned up

2016-04-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts reassigned YARN-4924: Assignee: Nathan Roberts > NM recovery race can lead to container not cleaned up >

[jira] [Commented] (YARN-4924) NM recovery race can lead to container not cleaned up

2016-04-05 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226956#comment-15226956 ] Nathan Roberts commented on YARN-4924: -- Observed the following race with NM recovery. 1)

[jira] [Created] (YARN-4924) NM recovery race can lead to container not cleaned up

2016-04-05 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-4924: Summary: NM recovery race can lead to container not cleaned up Key: YARN-4924 URL: https://issues.apache.org/jira/browse/YARN-4924 Project: Hadoop YARN

[jira] [Commented] (YARN-4834) ProcfsBasedProcessTree doesn't track daemonized processes

2016-04-05 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226424#comment-15226424 ] Nathan Roberts commented on YARN-4834: -- As a note, we were seeing this with slider applications. I

[jira] [Updated] (YARN-4834) ProcfsBasedProcessTree doesn't track daemonized processes

2016-04-05 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4834: - Attachment: YARN-4834.001.patch Simple fix that falls back to sessionID if process has become

[jira] [Commented] (YARN-4768) getAvailablePhysicalMemorySize can be inaccurate on linux

2016-03-19 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199732#comment-15199732 ] Nathan Roberts commented on YARN-4768: -- Any comments on this approach? >

[jira] [Created] (YARN-4834) ProcfsBasedProcessTree doesn't track daemonized processes

2016-03-19 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-4834: Summary: ProcfsBasedProcessTree doesn't track daemonized processes Key: YARN-4834 URL: https://issues.apache.org/jira/browse/YARN-4834 Project: Hadoop YARN

[jira] [Updated] (YARN-4768) getAvailablePhysicalMemorySize can be inaccurate on linux

2016-03-08 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4768: - Attachment: YARN-4768.patch Patch for trunk. Also changed getPhysicalMemorySize() to exclude: -

  1   2   >