[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.16.patch Fixing minor checkstyle issues > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Labels: oct16-hard > Attachments: YARN-4359.0.patch, YARN-4359.10.patch, > YARN-4359.11.patch, YARN-4359.12.patch, YARN-4359.13.patch, > YARN-4359.14.patch, YARN-4359.15.patch, YARN-4359.16.patch, > YARN-4359.3.patch, YARN-4359.4.patch, YARN-4359.5.patch, YARN-4359.6.patch, > YARN-4359.7.patch, YARN-4359.8.patch, YARN-4359.9.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.15.patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Labels: oct16-hard > Attachments: YARN-4359.0.patch, YARN-4359.10.patch, > YARN-4359.11.patch, YARN-4359.12.patch, YARN-4359.13.patch, > YARN-4359.14.patch, YARN-4359.15.patch, YARN-4359.3.patch, YARN-4359.4.patch, > YARN-4359.5.patch, YARN-4359.6.patch, YARN-4359.7.patch, YARN-4359.8.patch, > YARN-4359.9.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.14.patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Labels: oct16-hard > Attachments: YARN-4359.0.patch, YARN-4359.10.patch, > YARN-4359.11.patch, YARN-4359.12.patch, YARN-4359.13.patch, > YARN-4359.14.patch, YARN-4359.3.patch, YARN-4359.4.patch, YARN-4359.5.patch, > YARN-4359.6.patch, YARN-4359.7.patch, YARN-4359.8.patch, YARN-4359.9.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.13.patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Labels: oct16-hard > Attachments: YARN-4359.0.patch, YARN-4359.10.patch, > YARN-4359.11.patch, YARN-4359.12.patch, YARN-4359.13.patch, > YARN-4359.3.patch, YARN-4359.4.patch, YARN-4359.5.patch, YARN-4359.6.patch, > YARN-4359.7.patch, YARN-4359.8.patch, YARN-4359.9.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.12.patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Labels: oct16-hard > Attachments: YARN-4359.0.patch, YARN-4359.10.patch, > YARN-4359.11.patch, YARN-4359.12.patch, YARN-4359.3.patch, YARN-4359.4.patch, > YARN-4359.5.patch, YARN-4359.6.patch, YARN-4359.7.patch, YARN-4359.8.patch, > YARN-4359.9.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.10.patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Attachments: YARN-4359.0.patch, YARN-4359.10.patch, > YARN-4359.3.patch, YARN-4359.4.patch, YARN-4359.5.patch, YARN-4359.6.patch, > YARN-4359.7.patch, YARN-4359.8.patch, YARN-4359.9.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.9.patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Attachments: YARN-4359.0.patch, YARN-4359.3.patch, YARN-4359.4.patch, > YARN-4359.5.patch, YARN-4359.6.patch, YARN-4359.7.patch, YARN-4359.8.patch, > YARN-4359.9.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.8.patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Attachments: YARN-4359.0.patch, YARN-4359.3.patch, YARN-4359.4.patch, > YARN-4359.5.patch, YARN-4359.6.patch, YARN-4359.7.patch, YARN-4359.8.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.7.patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Attachments: YARN-4359.0.patch, YARN-4359.3.patch, YARN-4359.4.patch, > YARN-4359.5.patch, YARN-4359.6.patch, YARN-4359.7.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.6.patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Attachments: YARN-4359.0.patch, YARN-4359.3.patch, YARN-4359.4.patch, > YARN-4359.5.patch, YARN-4359.6.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.5.patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Attachments: YARN-4359.0.patch, YARN-4359.3.patch, YARN-4359.4.patch, > YARN-4359.5.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.4.patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Attachments: YARN-4359.0.patch, YARN-4359.3.patch, YARN-4359.4.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: (was: YARN-4359.2.patch) > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Attachments: YARN-4359.0.patch, YARN-4359.3.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.3.patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Attachments: YARN-4359.0.patch, YARN-4359.3.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.2.patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Attachments: YARN-4359.0.patch, YARN-4359.2.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15166439#comment-15166439 ] Ishai Menache commented on YARN-4359: - Will add additional tests soon. > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Attachments: YARN-4359.0.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[ https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4359: Attachment: YARN-4359.0.patch first version of the patch > Update LowCost agents logic to take advantage of YARN-4358 > -- > > Key: YARN-4359 > URL: https://issues.apache.org/jira/browse/YARN-4359 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Ishai Menache > Attachments: YARN-4359.0.patch > > > Given the improvements of YARN-4358, the LowCost agent should be improved to > leverage this, and operate on RLESparseResourceAllocation (ideally leveraging > the improvements of YARN-3454 to compute avaialable resources) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4525) Bug in RLESparseResourceAllocation.getRangeOverlapping(...)
[ https://issues.apache.org/jira/browse/YARN-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143392#comment-15143392 ] Ishai Menache commented on YARN-4525: - Added the patch that fixes this issue. [~curino] please review > Bug in RLESparseResourceAllocation.getRangeOverlapping(...) > --- > > Key: YARN-4525 > URL: https://issues.apache.org/jira/browse/YARN-4525 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ishai Menache >Assignee: Ishai Menache > Attachments: YARN-4525.patch > > > One of our tests detected a corner case in getRangeOverlapping: When the > RLESparseResourceAllocation object is a result of a merge operation, the > underlying map is a "view" within some range. If 'end' is outside that > range, headMap(..) throws an uncaught exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4525) Bug in RLESparseResourceAllocation.getRangeOverlapping(...)
[ https://issues.apache.org/jira/browse/YARN-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-4525: Attachment: YARN-4525.patch > Bug in RLESparseResourceAllocation.getRangeOverlapping(...) > --- > > Key: YARN-4525 > URL: https://issues.apache.org/jira/browse/YARN-4525 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ishai Menache >Assignee: Ishai Menache > Attachments: YARN-4525.patch > > > One of our tests detected a corner case in getRangeOverlapping: When the > RLESparseResourceAllocation object is a result of a merge operation, the > underlying map is a "view" within some range. If 'end' is outside that > range, headMap(..) throws an uncaught exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-4525) Bug in RLESparseResourceAllocation.getRangeOverlapping(...)
[ https://issues.apache.org/jira/browse/YARN-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache reassigned YARN-4525: --- Assignee: Ishai Menache (was: Carlo Curino) > Bug in RLESparseResourceAllocation.getRangeOverlapping(...) > --- > > Key: YARN-4525 > URL: https://issues.apache.org/jira/browse/YARN-4525 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ishai Menache >Assignee: Ishai Menache > > One of our tests detected a corner case in getRangeOverlapping: When the > RLESparseResourceAllocation object is a result of a merge operation, the > underlying map is a "view" within some range. If 'end' is outside that > range, headMap(..) throws an uncaught exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4525) Bug in RLESparseResourceAllocation.getRangeOverlapping(...)
Ishai Menache created YARN-4525: --- Summary: Bug in RLESparseResourceAllocation.getRangeOverlapping(...) Key: YARN-4525 URL: https://issues.apache.org/jira/browse/YARN-4525 Project: Hadoop YARN Issue Type: Bug Reporter: Ishai Menache Assignee: Carlo Curino One of our tests detected a corner case in getRangeOverlapping: When the RLESparseResourceAllocation object is a result of a merge operation, the underlying map is a "view" within some range. If 'end' is outside that range, headMap(..) throws an uncaught exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4525) Bug in RLESparseResourceAllocation.getRangeOverlapping(...)
[
https://issues.apache.org/jira/browse/YARN-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074355#comment-15074355
]
Ishai Menache commented on YARN-4525:
-
We suggest the following fix:
current code:
a = a.headMap(end, true);
fix:
if (end < a.lastKey()) {
a = a.headMap(end, true);
}
Thanks to [~jyaniv] for helping debug this issue.
> Bug in RLESparseResourceAllocation.getRangeOverlapping(...)
> ---
>
> Key: YARN-4525
> URL: https://issues.apache.org/jira/browse/YARN-4525
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ishai Menache
>Assignee: Carlo Curino
>
> One of our tests detected a corner case in getRangeOverlapping: When the
> RLESparseResourceAllocation object is a result of a merge operation, the
> underlying map is a "view" within some range. If 'end' is outside that
> range, headMap(..) throws an uncaught exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3656) LowCost: A Cost-Based Placement Agent for YARN Reservations
[
https://issues.apache.org/jira/browse/YARN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627082#comment-14627082
]
Ishai Menache commented on YARN-3656:
-
Thank you [~asuresh] for your comments!
We will remove the redundant “if” conditions from TryManyReservationAgents.java.
Regarding your suggestion of incorporating a fallback strategy for
ReservationAgent – it is an interesting idea, but we believe it might lead to a
couple of issues.
The behavior of TryManyReservationAgents is not straightforward. For example,
if you have an agent defined by RA = TryManyReservationAgents{ RA1, RA2, RA3 },
the overall performance of RA depends on how the algorithms RA1, RA2 and RA3
affect one another, and this dependency can be complex. In general, you might
want to implement several TryMany policies with different fallback policies,
depending on the available capacity, job characteristics etc. This is why we
preferred to implement the collection of the algorithms as a LinkedList, which
gives full flexibility to the user. On a related note, the fallback setting
might accidentally end up in a cyclic dependency between the different
algorithms. Please let us know if you agree.
LowCost: A Cost-Based Placement Agent for YARN Reservations
---
Key: YARN-3656
URL: https://issues.apache.org/jira/browse/YARN-3656
Project: Hadoop YARN
Issue Type: Improvement
Components: capacityscheduler, resourcemanager
Affects Versions: 2.6.0
Reporter: Ishai Menache
Assignee: Jonathan Yaniv
Labels: capacity-scheduler, resourcemanager
Attachments: LowCostRayonExternal.pdf, YARN-3656-v1.1.patch,
YARN-3656-v1.2.patch, YARN-3656-v1.patch, lowcostrayonexternal_v2.pdf
YARN-1051 enables SLA support by allowing users to reserve cluster capacity
ahead of time. YARN-1710 introduced a greedy agent for placing user
reservations. The greedy agent makes fast placement decisions but at the cost
of ignoring the cluster committed resources, which might result in blocking
the cluster resources for certain periods of time, and in turn rejecting some
arriving jobs.
We propose LowCost – a new cost-based planning algorithm. LowCost “spreads”
the demand of the job throughout the allowed time-window according to a
global, load-based cost function.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3656) LowCost: A Cost-Based Placement Agent for YARN Reservations
[ https://issues.apache.org/jira/browse/YARN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-3656: Attachment: YARN-3656-v1.patch LowCost: A Cost-Based Placement Agent for YARN Reservations --- Key: YARN-3656 URL: https://issues.apache.org/jira/browse/YARN-3656 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler, resourcemanager Reporter: Ishai Menache Attachments: LowCostRayonExternal.pdf, YARN-3656-v1.patch YARN-1051 enables SLA support by allowing users to reserve cluster capacity ahead of time. YARN-1710 introduced a greedy agent for placing user reservations. The greedy agent makes fast placement decisions but at the cost of ignoring the cluster committed resources, which might result in blocking the cluster resources for certain periods of time, and in turn rejecting some arriving jobs. We propose LowCost – a new cost-based planning algorithm. LowCost “spreads” the demand of the job throughout the allowed time-window according to a global, load-based cost function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3656) LowCost: A Cost-Based Placement Agent for YARN Reservations
[ https://issues.apache.org/jira/browse/YARN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-3656: Attachment: lowcostrayonexternal_v2.pdf new version of the design doc, including a class diagram. LowCost: A Cost-Based Placement Agent for YARN Reservations --- Key: YARN-3656 URL: https://issues.apache.org/jira/browse/YARN-3656 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler, resourcemanager Affects Versions: 2.6.0 Reporter: Ishai Menache Assignee: Jonathan Yaniv Labels: capacity-scheduler, resourcemanager Attachments: LowCostRayonExternal.pdf, YARN-3656-v1.patch, lowcostrayonexternal_v2.pdf YARN-1051 enables SLA support by allowing users to reserve cluster capacity ahead of time. YARN-1710 introduced a greedy agent for placing user reservations. The greedy agent makes fast placement decisions but at the cost of ignoring the cluster committed resources, which might result in blocking the cluster resources for certain periods of time, and in turn rejecting some arriving jobs. We propose LowCost – a new cost-based planning algorithm. LowCost “spreads” the demand of the job throughout the allowed time-window according to a global, load-based cost function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3656) LowCost: A Cost-Based Placement Agent for YARN Reservations
Ishai Menache created YARN-3656: --- Summary: LowCost: A Cost-Based Placement Agent for YARN Reservations Key: YARN-3656 URL: https://issues.apache.org/jira/browse/YARN-3656 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler, resourcemanager Reporter: Ishai Menache YARN-1051 enables SLA support by allowing users to reserve cluster capacity ahead of time. YARN-1710 introduced a greedy agent for placing user reservations. The greedy agent makes fast placement decisions but at the cost of ignoring the cluster committed resources, which might result in blocking the cluster resources for certain periods of time, and in turn rejecting some arriving jobs. We propose LowCost – a new cost-based planning algorithm. LowCost “spreads” the demand of the job throughout the allowed time-window according to a global, load-based cost function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3656) LowCost: A Cost-Based Placement Agent for YARN Reservations
[ https://issues.apache.org/jira/browse/YARN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishai Menache updated YARN-3656: Attachment: LowCostRayonExternal.pdf This tech-report summarizes the details of LowCost, as well as our experimental results which show benefits in using LowCost on a variety of performance metrics LowCost: A Cost-Based Placement Agent for YARN Reservations --- Key: YARN-3656 URL: https://issues.apache.org/jira/browse/YARN-3656 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler, resourcemanager Reporter: Ishai Menache Attachments: LowCostRayonExternal.pdf YARN-1051 enables SLA support by allowing users to reserve cluster capacity ahead of time. YARN-1710 introduced a greedy agent for placing user reservations. The greedy agent makes fast placement decisions but at the cost of ignoring the cluster committed resources, which might result in blocking the cluster resources for certain periods of time, and in turn rejecting some arriving jobs. We propose LowCost – a new cost-based planning algorithm. LowCost “spreads” the demand of the job throughout the allowed time-window according to a global, load-based cost function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3656) LowCost: A Cost-Based Placement Agent for YARN Reservations
[ https://issues.apache.org/jira/browse/YARN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14546361#comment-14546361 ] Ishai Menache commented on YARN-3656: - LowCost judiciously “spreads” the demand of the job throughout the allowed time-window according to a global, load-based cost function. This leads to more balanced allocations, and in turn substantially improves the acceptance rate of jobs and the cluster utilization. LowCost: A Cost-Based Placement Agent for YARN Reservations --- Key: YARN-3656 URL: https://issues.apache.org/jira/browse/YARN-3656 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler, resourcemanager Reporter: Ishai Menache YARN-1051 enables SLA support by allowing users to reserve cluster capacity ahead of time. YARN-1710 introduced a greedy agent for placing user reservations. The greedy agent makes fast placement decisions but at the cost of ignoring the cluster committed resources, which might result in blocking the cluster resources for certain periods of time, and in turn rejecting some arriving jobs. We propose LowCost – a new cost-based planning algorithm. LowCost “spreads” the demand of the job throughout the allowed time-window according to a global, load-based cost function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
