[jira] [Commented] (MESOS-9806) Address allocator performance regression due to the removal of quota role sorter.

2019-08-22 Thread Benjamin Mahler (Jira)


[ 
https://issues.apache.org/jira/browse/MESOS-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913777#comment-16913777
 ] 

Benjamin Mahler commented on MESOS-9806:


{noformat}
commit de90b2b3078e06975ab2061db821cfe7dda8
Author: Benjamin Mahler 
Date:   Thu Aug 22 17:41:28 2019 -0400

Optimized Resources::shrink.

Master:
*HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
Made 3500 allocations in 30.37 secs
Made 0 allocation in 27.05 secs

Master + this patch:
*HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
Made 3500 allocations in 24.15 secs
Made 0 allocation in 20.48 secs

Review: https://reviews.apache.org/r/71353
{noformat}

{noformat}
commit 05e5ca4b3446e34447f632463efe9a34b4bace7f
Author: Benjamin Mahler 
Date:   Thu Aug 22 17:42:57 2019 -0400

Added ResourceQuantities::fromScalarResource.

Master + previous patches:
*HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
Made 3500 allocations in 24.15 secs
Made 0 allocation in 20.48 secs

Master + previous patches + this patch:
Master + this patch:
*HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
Made 3500 allocations in 23.37 secs
Made 0 allocation in 19.72 secs

Review: https://reviews.apache.org/r/71354
{noformat}

> Address allocator performance regression due to the removal of quota role 
> sorter.
> -
>
> Key: MESOS-9806
> URL: https://issues.apache.org/jira/browse/MESOS-9806
> Project: Mesos
>  Issue Type: Improvement
>  Components: allocation
>Reporter: Meng Zhu
>Assignee: Meng Zhu
>Priority: Critical
>  Labels: resource-management
>
> In MESOS-9802, we removed the quota role sorter which is tech debt.
> However, this slows down the allocator. The problem is that in the first 
> stage, even though a cluster might have no active roles with non-default 
> quota, the allocator will now have to sort and go through each and every role 
> in the cluster. Benchmark result shows that for 1k roles with 2k frameworks, 
> the allocator could experience ~50% performance degradation.
> There are a couple of ways to address this issue. For example, we could make 
> the sorter aware of quota. And add a method, say `sortQuotaRoles`, to return 
> all the roles with non-default quota. Alternatively, an even better approach 
> would be to deprecate the sorter concept and just have two standalone 
> functions e.g. sortRoles() and sortQuotaRoles() that takes in the role tree 
> structure (not yet exist in the allocator) and return the sorted roles.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (MESOS-9806) Address allocator performance regression due to the removal of quota role sorter.

2019-08-22 Thread Benjamin Mahler (Jira)


[ 
https://issues.apache.org/jira/browse/MESOS-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913709#comment-16913709
 ] 

Benjamin Mahler commented on MESOS-9806:


{noformat}
commit b6c87d7c44346b2497ace65b1d2060ee423aa772
Author: Benjamin Mahler 
Date:   Wed Aug 21 20:10:42 2019 -0400

Eliminated double lookups in the allocator.

Review: https://reviews.apache.org/r/71345
{noformat}

{noformat}
commit 790c4e72e1460035b13bf27f2cb8999709e9767e
Author: Benjamin Mahler 
Date:   Wed Aug 21 20:11:31 2019 -0400

Avoid duplicate allocatableTo call in the allocator.

Review: https://reviews.apache.org/r/71346
{noformat}

> Address allocator performance regression due to the removal of quota role 
> sorter.
> -
>
> Key: MESOS-9806
> URL: https://issues.apache.org/jira/browse/MESOS-9806
> Project: Mesos
>  Issue Type: Improvement
>  Components: allocation
>Reporter: Meng Zhu
>Assignee: Meng Zhu
>Priority: Critical
>  Labels: resource-management
>
> In MESOS-9802, we removed the quota role sorter which is tech debt.
> However, this slows down the allocator. The problem is that in the first 
> stage, even though a cluster might have no active roles with non-default 
> quota, the allocator will now have to sort and go through each and every role 
> in the cluster. Benchmark result shows that for 1k roles with 2k frameworks, 
> the allocator could experience ~50% performance degradation.
> There are a couple of ways to address this issue. For example, we could make 
> the sorter aware of quota. And add a method, say `sortQuotaRoles`, to return 
> all the roles with non-default quota. Alternatively, an even better approach 
> would be to deprecate the sorter concept and just have two standalone 
> functions e.g. sortRoles() and sortQuotaRoles() that takes in the role tree 
> structure (not yet exist in the allocator) and return the sorted roles.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (MESOS-9806) Address allocator performance regression due to the removal of quota role sorter.

2019-08-21 Thread Benjamin Mahler (Jira)


[ 
https://issues.apache.org/jira/browse/MESOS-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912823#comment-16912823
 ] 

Benjamin Mahler commented on MESOS-9806:


https://reviews.apache.org/r/71345/
https://reviews.apache.org/r/71346/
https://reviews.apache.org/r/71347/

Master branch:
{noformat}
HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
Made 3500 allocations in 36.1185929secs
Made 0 allocation in 32.62218602secs
{noformat}

After all patches in review chain:
{noformat}
HierarchicalAllocator_WithQuotaParam.LargeAndSmallQuota/2
Made 3500 allocations in 21.389381617secs
Made 0 allocation in 18.593000222secs
{noformat}

> Address allocator performance regression due to the removal of quota role 
> sorter.
> -
>
> Key: MESOS-9806
> URL: https://issues.apache.org/jira/browse/MESOS-9806
> Project: Mesos
>  Issue Type: Improvement
>  Components: allocation
>Reporter: Meng Zhu
>Assignee: Meng Zhu
>Priority: Critical
>  Labels: resource-management
>
> In MESOS-9802, we removed the quota role sorter which is tech debt.
> However, this slows down the allocator. The problem is that in the first 
> stage, even though a cluster might have no active roles with non-default 
> quota, the allocator will now have to sort and go through each and every role 
> in the cluster. Benchmark result shows that for 1k roles with 2k frameworks, 
> the allocator could experience ~50% performance degradation.
> There are a couple of ways to address this issue. For example, we could make 
> the sorter aware of quota. And add a method, say `sortQuotaRoles`, to return 
> all the roles with non-default quota. Alternatively, an even better approach 
> would be to deprecate the sorter concept and just have two standalone 
> functions e.g. sortRoles() and sortQuotaRoles() that takes in the role tree 
> structure (not yet exist in the allocator) and return the sorted roles.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)