[jira] [Created] (YARN-2410) Nodemanager ShuffleHandler can easily exhaust file descriptors

2014-08-12 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-2410: Summary: Nodemanager ShuffleHandler can easily exhaust file descriptors Key: YARN-2410 URL: https://issues.apache.org/jira/browse/YARN-2410 Project: Hadoop YARN

[jira] [Commented] (YARN-2056) Disable preemption at Queue level

2014-08-14 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14097823#comment-14097823 ] Nathan Roberts commented on YARN-2056: -- [~sunilg] I'm not following the doubt. It may

[jira] [Commented] (YARN-2431) NM restart: cgroup is not removed for reacquired containers

2014-08-21 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106104#comment-14106104 ] Nathan Roberts commented on YARN-2431: -- +1 lgtm, non-binding NM restart: cgroup is

[jira] [Commented] (YARN-2440) Cgroups should limit YARN containers to cores allocated in yarn-site.xml

2014-08-22 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106889#comment-14106889 ] Nathan Roberts commented on YARN-2440: -- Thanks Varun for the patch. I'm wondering if

[jira] [Created] (YARN-139) Interrupted Exception within AsyncDispatcher leads to user confusion

2012-10-01 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-139: --- Summary: Interrupted Exception within AsyncDispatcher leads to user confusion Key: YARN-139 URL: https://issues.apache.org/jira/browse/YARN-139 Project: Hadoop YARN

[jira] [Created] (YARN-162) nodemanager log aggregation has scaling issues with namenode

2012-10-16 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-162: --- Summary: nodemanager log aggregation has scaling issues with namenode Key: YARN-162 URL: https://issues.apache.org/jira/browse/YARN-162 Project: Hadoop YARN

[jira] [Created] (YARN-212) NM state machine ignores an APPLICATION_CONTAINER_FINISHED event when it shouldn't

2012-11-12 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-212: --- Summary: NM state machine ignores an APPLICATION_CONTAINER_FINISHED event when it shouldn't Key: YARN-212 URL: https://issues.apache.org/jira/browse/YARN-212 Project:

[jira] [Commented] (YARN-212) NM state machine ignores an APPLICATION_CONTAINER_FINISHED event when it shouldn't

2012-11-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495622#comment-13495622 ] Nathan Roberts commented on YARN-212: - The interesting parts of the logs are:

[jira] [Updated] (YARN-212) NM state machine ignores an APPLICATION_CONTAINER_FINISHED event when it shouldn't

2012-11-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-212: Attachment: yarn-212.txt NM state machine ignores an APPLICATION_CONTAINER_FINISHED event when

[jira] [Updated] (YARN-212) NM state machine ignores an APPLICATION_CONTAINER_FINISHED event when it shouldn't

2012-11-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-212: Attachment: yarn-212.txt Fixed timing issue in TestLogAggregationService NM state

[jira] [Commented] (YARN-270) RM scheduler event handler thread gets behind

2012-12-17 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13534337#comment-13534337 ] Nathan Roberts commented on YARN-270: - Could we also add some additional flow control

[jira] [Created] (YARN-520) webservices API ws/v1/cluster/nodes doesn't return LOST nodes

2013-03-29 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-520: --- Summary: webservices API ws/v1/cluster/nodes doesn't return LOST nodes Key: YARN-520 URL: https://issues.apache.org/jira/browse/YARN-520 Project: Hadoop YARN

[jira] [Commented] (YARN-1912) ResourceLocalizer started without any jvm memory control

2014-04-08 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963010#comment-13963010 ] Nathan Roberts commented on YARN-1912: -- Doesn't it default to MIN(25%_of_memory,1GB)?

[jira] [Commented] (YARN-1912) ResourceLocalizer started without any jvm memory control

2014-04-09 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964198#comment-13964198 ] Nathan Roberts commented on YARN-1912: -- Ah. Thanks Stanley for the pointer and

[jira] [Created] (YARN-1975) Used resources shows escaped html in scheduler page

2014-04-23 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-1975: Summary: Used resources shows escaped html in scheduler page Key: YARN-1975 URL: https://issues.apache.org/jira/browse/YARN-1975 Project: Hadoop YARN Issue

[jira] [Updated] (YARN-1975) Used resources shows escaped html in scheduler page

2014-04-23 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-1975: - Description: Used resources displays as amp;lt;memory:, vCores;amp;gt; with capacity

[jira] [Commented] (YARN-322) Add cpu information to queue metrics

2014-05-13 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995093#comment-13995093 ] Nathan Roberts commented on YARN-322: - Arun, does this patch address what you were

[jira] [Commented] (YARN-322) Add cpu information to queue metrics

2014-05-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992751#comment-13992751 ] Nathan Roberts commented on YARN-322: - TestRMRestart failure isn't related to this

[jira] [Created] (YARN-2072) RM/NM UIs and webservices are missing vcore information

2014-05-19 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-2072: Summary: RM/NM UIs and webservices are missing vcore information Key: YARN-2072 URL: https://issues.apache.org/jira/browse/YARN-2072 Project: Hadoop YARN

[jira] [Updated] (YARN-2072) RM/NM UIs and webservices are missing vcore information

2014-05-30 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-2072: - Attachment: YARN-2072.patch RM/NM UIs and webservices are missing vcore information

[jira] [Commented] (YARN-2056) Disable preemption at Queue level

2014-06-11 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14028392#comment-14028392 ] Nathan Roberts commented on YARN-2056: -- Could this be accomplished by changing

[jira] [Updated] (YARN-2072) RM/NM UIs and webservices are missing vcore information

2014-06-23 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-2072: - Attachment: YARN-2072.patch Thanks for the review Tom! I fixed the getReservedVirtualCores() bug

[jira] [Created] (YARN-2809) Implement workaround for linux kernel panic when removing cgroup

2014-11-05 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-2809: Summary: Implement workaround for linux kernel panic when removing cgroup Key: YARN-2809 URL: https://issues.apache.org/jira/browse/YARN-2809 Project: Hadoop YARN

[jira] [Commented] (YARN-2809) Implement workaround for linux kernel panic when removing cgroup

2014-11-05 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14198560#comment-14198560 ] Nathan Roberts commented on YARN-2809: -- Stack trace: {noformat} [8150d4a8] ?

[jira] [Updated] (YARN-2809) Implement workaround for linux kernel panic when removing cgroup

2014-11-25 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-2809: - Attachment: YARN-2809.patch Implement workaround for linux kernel panic when removing cgroup

[jira] [Created] (YARN-2904) Use linux cgroups to enhance container tear down

2014-11-25 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-2904: Summary: Use linux cgroups to enhance container tear down Key: YARN-2904 URL: https://issues.apache.org/jira/browse/YARN-2904 Project: Hadoop YARN Issue

[jira] [Updated] (YARN-2809) Implement workaround for linux kernel panic when removing cgroup

2015-02-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-2809: - Attachment: YARN-2809-v2.patch upmerge to latest trunk Implement workaround for linux kernel

[jira] [Commented] (YARN-3298) User-limit should be enforced in CapacityScheduler

2015-03-09 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353053#comment-14353053 ] Nathan Roberts commented on YARN-3298: -- Thanks [~leftnoteasy] for the additional

[jira] [Created] (YARN-3309) Capacity scheduler can wait a very long time for node locality

2015-03-09 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-3309: Summary: Capacity scheduler can wait a very long time for node locality Key: YARN-3309 URL: https://issues.apache.org/jira/browse/YARN-3309 Project: Hadoop YARN

[jira] [Commented] (YARN-1963) Support priorities across applications within the same queue

2015-03-09 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353677#comment-14353677 ] Nathan Roberts commented on YARN-1963: -- {quote} Without some sort of labels, it will

[jira] [Commented] (YARN-3215) Respect labels in CapacityScheduler when computing headroom

2015-03-09 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353800#comment-14353800 ] Nathan Roberts commented on YARN-3215: -- Hi [~leftnoteasy]. Can you provide a summary

[jira] [Commented] (YARN-3298) User-limit should be enforced in CapacityScheduler

2015-03-09 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353833#comment-14353833 ] Nathan Roberts commented on YARN-3298: -- [~leftnoteasy], won't that be extremely close

[jira] [Commented] (YARN-3298) User-limit should be enforced in CapacityScheduler

2015-03-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350614#comment-14350614 ] Nathan Roberts commented on YARN-3298: -- Hi Wangda. I'm a little concerned about this

[jira] [Commented] (YARN-3298) User-limit should be enforced in CapacityScheduler

2015-03-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358708#comment-14358708 ] Nathan Roberts commented on YARN-3298: -- I agree. Let's not change anything for the

[jira] [Commented] (YARN-3298) User-limit should be enforced in CapacityScheduler

2015-03-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355030#comment-14355030 ] Nathan Roberts commented on YARN-3298: -- If you have a prototype patch, please post it

[jira] [Commented] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit

2015-03-31 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389550#comment-14389550 ] Nathan Roberts commented on YARN-3388: -- [~leftnoteasy] - Thanks for the comments. I'm

[jira] [Created] (YARN-3388) userlimit isn't playing well with DRF calculator

2015-03-23 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-3388: Summary: userlimit isn't playing well with DRF calculator Key: YARN-3388 URL: https://issues.apache.org/jira/browse/YARN-3388 Project: Hadoop YARN Issue

[jira] [Commented] (YARN-3388) userlimit isn't playing well with DRF calculator

2015-03-23 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376060#comment-14376060 ] Nathan Roberts commented on YARN-3388: -- Example (lots of things going on in this

[jira] [Updated] (YARN-3388) userlimit isn't playing well with DRF calculator

2015-03-26 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-3388: - Attachment: YARN-3388-v0.patch Initial patch for comments on approach. Seems to work well in basic

[jira] [Updated] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit

2015-04-02 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-3388: - Attachment: YARN-3388-v1.patch Hi [~leftnoteasy]. Uploaded a new version of patch that addresses

[jira] [Commented] (YARN-3361) CapacityScheduler side changes to support non-exclusive node labels

2015-04-13 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492951#comment-14492951 ] Nathan Roberts commented on YARN-3361: -- Hi [~leftnoteasy]. One comment/question.

[jira] [Commented] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit

2015-04-13 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493020#comment-14493020 ] Nathan Roberts commented on YARN-3388: -- Thanks [~leftnoteasy] for the comments.

[jira] [Commented] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit

2015-05-04 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526763#comment-14526763 ] Nathan Roberts commented on YARN-3388: -- Yes. I have a patch which I think is close. I

[jira] [Updated] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit

2015-05-14 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-3388: - Attachment: YARN-3388-v2.patch [~leftnoteasy], please take a look at this version of the patch.

[jira] [Commented] (YARN-3361) CapacityScheduler side changes to support non-exclusive node labels

2015-04-14 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494490#comment-14494490 ] Nathan Roberts commented on YARN-3361: -- A follow-up jira is ok. We can take care of it

[jira] [Commented] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit

2015-04-08 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485413#comment-14485413 ] Nathan Roberts commented on YARN-3388: -- Test failures don't appear related to patch.

[jira] [Commented] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit

2015-05-20 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553165#comment-14553165 ] Nathan Roberts commented on YARN-3388: -- Thanks [~leftnoteasy] for the comments. I

[jira] [Commented] (YARN-3945) maxApplicationsPerUser is wrongly calculated

2015-07-30 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648418#comment-14648418 ] Nathan Roberts commented on YARN-3945: -- Hi [~leftnoteasy]. Regarding

[jira] [Commented] (YARN-3945) maxApplicationsPerUser is wrongly calculated

2015-07-30 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648032#comment-14648032 ] Nathan Roberts commented on YARN-3945: -- bq. Though the class doc(ActiveUsersManager)

[jira] [Created] (YARN-4052) Set SO_KEEPALIVE on NM servers

2015-08-13 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-4052: Summary: Set SO_KEEPALIVE on NM servers Key: YARN-4052 URL: https://issues.apache.org/jira/browse/YARN-4052 Project: Hadoop YARN Issue Type: Bug

[jira] [Commented] (YARN-3945) maxApplicationsPerUser is wrongly calculated

2015-07-21 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14635246#comment-14635246 ] Nathan Roberts commented on YARN-3945: -- My feeling is the documentation on

[jira] [Updated] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-10-23 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4287: - Attachment: YARN-4287-v3.patch V3 of patch. Thanks again for the comments. bq.

[jira] [Commented] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-10-23 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971176#comment-14971176 ] Nathan Roberts commented on YARN-4287: -- Thanks for the comments. You're right that the logic can be

[jira] [Updated] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-10-22 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4287: - Attachment: YARN-4287-v2.patch Fixed unit test failures and addressed most checkstyle errors >

[jira] [Commented] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-10-29 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980503#comment-14980503 ] Nathan Roberts commented on YARN-4287: -- +1 on percentages. My only concern is that node-locality-delay

[jira] [Updated] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-10-27 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4287: - Attachment: YARN-4287-v4.patch V4 of patch. - I moved the calculation of locality delays out of

[jira] [Updated] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-10-28 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4287: - Attachment: YARN-4287-minimal.patch [~leftnoteasy], Another very simple approach is to just not

[jira] [Commented] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-10-26 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974512#comment-14974512 ] Nathan Roberts commented on YARN-4287: -- Thanks [~leftnoteasy] for the comments. {quote} 2.

[jira] [Created] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-10-21 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-4287: Summary: Capacity Scheduler: Rack Locality improvement Key: YARN-4287 URL: https://issues.apache.org/jira/browse/YARN-4287 Project: Hadoop YARN Issue Type:

[jira] [Updated] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-10-21 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4287: - Attachment: YARN-4287.patch > Capacity Scheduler: Rack Locality improvement >

[jira] [Commented] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-10-27 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976386#comment-14976386 ] Nathan Roberts commented on YARN-4287: -- Thanks [~leftnoteasy] for the quick responses. {quote} I

[jira] [Updated] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-11-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4287: - Attachment: YARN-4287-minimal-v4.patch Thanks [~leftnoteasy] for the comments. Made the following

[jira] [Updated] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-11-09 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4287: - Attachment: YARN-4287-minimal-v3.patch Noticed simple spelling error > Capacity Scheduler: Rack

[jira] [Commented] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-11-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003187#comment-15003187 ] Nathan Roberts commented on YARN-4287: -- I will put up a 2.7 version tomorrow morning. > Capacity

[jira] [Updated] (YARN-4287) Capacity Scheduler: Rack Locality improvement

2015-11-13 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4287: - Attachment: YARN-4287-minimal-v4-branch-2.7.patch 2.7 version of patch. > Capacity Scheduler:

[jira] [Commented] (YARN-2410) Nodemanager ShuffleHandler can possible exhaust file descriptors

2015-09-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738773#comment-14738773 ] Nathan Roberts commented on YARN-2410: -- Thanks for the additional code comments. +1 > Nodemanager

[jira] [Commented] (YARN-2410) Nodemanager ShuffleHandler can possible exhaust file descriptors

2015-09-09 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737654#comment-14737654 ] Nathan Roberts commented on YARN-2410: -- One minor comment. If it's not too much trouble, could you add

[jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers

2016-01-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085677#comment-15085677 ] Nathan Roberts commented on YARN-1011: -- bq. This is one of the reasons I was proposing the notion of a

[jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers

2016-01-05 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15083880#comment-15083880 ] Nathan Roberts commented on YARN-1011: -- Very excited about this feature and agree that we should make

[jira] [Commented] (YARN-5214) Pending on synchronized method DirectoryCollection#checkDirs can hang NM's NodeStatusUpdater

2016-06-08 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321342#comment-15321342 ] Nathan Roberts commented on YARN-5214: -- I'm not suggesting this change shouldn't be made but keep in

[jira] [Created] (YARN-5202) Dynamic Overcommit of Node Resources - POC

2016-06-06 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-5202: Summary: Dynamic Overcommit of Node Resources - POC Key: YARN-5202 URL: https://issues.apache.org/jira/browse/YARN-5202 Project: Hadoop YARN Issue Type:

[jira] [Updated] (YARN-5202) Dynamic Overcommit of Node Resources - POC

2016-06-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-5202: - Attachment: YARN-5202.patch Originally branched from commit:

[jira] [Commented] (YARN-5214) Pending on synchronized method DirectoryCollection#checkDirs can hang NM's NodeStatusUpdater

2016-06-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324682#comment-15324682 ] Nathan Roberts commented on YARN-5214: -- [~djp]. I agree it makes sense to keep the heartbeat path as

[jira] [Updated] (YARN-3388) Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit

2016-06-14 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-3388: - Attachment: YARN-3388-v3.patch [~leftnoteasy], [~eepayne]. Ok, "soon" was extremely relative;)

[jira] [Commented] (YARN-5215) Scheduling containers based on external load in the servers

2016-06-15 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332601#comment-15332601 ] Nathan Roberts commented on YARN-5215: -- Thanks [~elgoiri] for the work. Maybe Summit would be a good

[jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers

2016-01-13 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096938#comment-15096938 ] Nathan Roberts commented on YARN-1011: -- Thanks for the update [~kasha]. I have a few questions but

[jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers

2016-01-13 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097040#comment-15097040 ] Nathan Roberts commented on YARN-1011: -- Won't the scheduler just try to assign it to that application

[jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers

2016-01-19 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107009#comment-15107009 ] Nathan Roberts commented on YARN-1011: -- bq. Welcome any thoughts/suggestions on handling promotion if

[jira] [Created] (YARN-4834) ProcfsBasedProcessTree doesn't track daemonized processes

2016-03-19 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-4834: Summary: ProcfsBasedProcessTree doesn't track daemonized processes Key: YARN-4834 URL: https://issues.apache.org/jira/browse/YARN-4834 Project: Hadoop YARN

[jira] [Commented] (YARN-4834) ProcfsBasedProcessTree doesn't track daemonized processes

2016-04-05 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226424#comment-15226424 ] Nathan Roberts commented on YARN-4834: -- As a note, we were seeing this with slider applications. I

[jira] [Updated] (YARN-4834) ProcfsBasedProcessTree doesn't track daemonized processes

2016-04-05 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4834: - Attachment: YARN-4834.001.patch Simple fix that falls back to sessionID if process has become

[jira] [Commented] (YARN-4924) NM recovery race can lead to container not cleaned up

2016-04-05 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15226956#comment-15226956 ] Nathan Roberts commented on YARN-4924: -- Observed the following race with NM recovery. 1)

[jira] [Created] (YARN-4924) NM recovery race can lead to container not cleaned up

2016-04-05 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-4924: Summary: NM recovery race can lead to container not cleaned up Key: YARN-4924 URL: https://issues.apache.org/jira/browse/YARN-4924 Project: Hadoop YARN

[jira] [Assigned] (YARN-4924) NM recovery race can lead to container not cleaned up

2016-04-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts reassigned YARN-4924: Assignee: Nathan Roberts > NM recovery race can lead to container not cleaned up >

[jira] [Commented] (YARN-4924) NM recovery race can lead to container not cleaned up

2016-04-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15228243#comment-15228243 ] Nathan Roberts commented on YARN-4924: -- Thanks [~sandflee], [~jlowe] for the suggestion. I'll work up

[jira] [Updated] (YARN-4924) NM recovery race can lead to container not cleaned up

2016-04-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4924: - Assignee: (was: Nathan Roberts) > NM recovery race can lead to container not cleaned up >

[jira] [Commented] (YARN-4924) NM recovery race can lead to container not cleaned up

2016-04-06 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15228258#comment-15228258 ] Nathan Roberts commented on YARN-4924: -- Sorry [~sandflee]. I missed your comment about updating

[jira] [Commented] (YARN-4768) getAvailablePhysicalMemorySize can be inaccurate on linux

2016-03-19 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199732#comment-15199732 ] Nathan Roberts commented on YARN-4768: -- Any comments on this approach? >

[jira] [Updated] (YARN-4768) getAvailablePhysicalMemorySize can be inaccurate on linux

2016-03-08 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4768: - Attachment: YARN-4768.patch Patch for trunk. Also changed getPhysicalMemorySize() to exclude: -

[jira] [Created] (YARN-4768) getAvailablePhysicalMemorySize can be inaccurate on linux

2016-03-04 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-4768: Summary: getAvailablePhysicalMemorySize can be inaccurate on linux Key: YARN-4768 URL: https://issues.apache.org/jira/browse/YARN-4768 Project: Hadoop YARN

[jira] [Commented] (YARN-4556) TestFifoScheduler.testResourceOverCommit fails

2016-04-21 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252593#comment-15252593 ] Nathan Roberts commented on YARN-4556: -- Patch seems like a reasonable test improvement. +1 non-binding

[jira] [Created] (YARN-5003) Add container resource to RM audit log

2016-04-27 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-5003: Summary: Add container resource to RM audit log Key: YARN-5003 URL: https://issues.apache.org/jira/browse/YARN-5003 Project: Hadoop YARN Issue Type:

[jira] [Updated] (YARN-5003) Add container resource to RM audit log

2016-04-27 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-5003: - Attachment: YARN-5003.001.patch Attaching patch > Add container resource to RM audit log >

[jira] [Commented] (YARN-5013) Allow applications to provide input on amount of locality delay to use

2016-04-28 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263106#comment-15263106 ] Nathan Roberts commented on YARN-5013: -- Re-posting latest comment from [~Naganarasimha] Thanks for

[jira] [Commented] (YARN-4963) capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable

2016-04-28 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263109#comment-15263109 ] Nathan Roberts commented on YARN-4963: -- Sorry it took so long to get back to this. I filed YARN-5013

[jira] [Created] (YARN-5013) Allow applications to provide input on amount of locality delay to use

2016-04-28 Thread Nathan Roberts (JIRA)
Nathan Roberts created YARN-5013: Summary: Allow applications to provide input on amount of locality delay to use Key: YARN-5013 URL: https://issues.apache.org/jira/browse/YARN-5013 Project: Hadoop

[jira] [Commented] (YARN-5008) LeveldbRMStateStore database can grow substantially leading to long recovery times

2016-04-28 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262472#comment-15262472 ] Nathan Roberts commented on YARN-5008: -- Thanks for the patch. LGTM. +1 non-binding >

[jira] [Commented] (YARN-5039) Applications ACCEPTED but not starting

2016-05-11 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280191#comment-15280191 ] Nathan Roberts commented on YARN-5039: -- Thanks [~milesc]. This seems to be an Amazon emr thing (unless

[jira] [Commented] (YARN-5039) Applications ACCEPTED but not starting

2016-05-10 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279018#comment-15279018 ] Nathan Roberts commented on YARN-5039: -- Thanks [~milesc]! Still not quite enough. How about

[jira] [Updated] (YARN-4963) capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable

2016-05-12 Thread Nathan Roberts (JIRA)
[ https://issues.apache.org/jira/browse/YARN-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Roberts updated YARN-4963: - Attachment: YARN-4963.003.patch Thank you [~leftnoteasy] for the comments! I have addressed them

  1   2   >