[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066508#comment-15066508
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/921


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064617#comment-15064617
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/921#issuecomment-165880798
  
@d2r thanks!

@revans2 @harshach @hustfxj @rfarivar @HeartSaVioR @knusbaum @redsanket do 
you have any additional concerns?


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064588#comment-15064588
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/921#issuecomment-165878491
  
All tests passed except jdk7 storm-kafka.  Created 
[STORM-1404](https://issues.apache.org/jira/browse/STORM-1404)

Great work. +1


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064527#comment-15064527
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/921#issuecomment-165874443
  
@d2r again thanks for your review.  I think I have addressed all of your 
comments


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064517#comment-15064517
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48058248
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064510#comment-15064510
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48058088
  
--- Diff: storm-core/src/jvm/backtype/storm/scheduler/resource/User.java ---
@@ -0,0 +1,345 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.TopologyDetails;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeSet;
+
+public class User {
+private String userId;
+//Topologies yet to be scheduled sorted by priority for each user
+private TreeSet pendingQueue = new 
TreeSet(new PQsortByPriorityAndSubmittionTime());
+
+//Topologies yet to be scheduled sorted by priority for each user
+private TreeSet runningQueue = new 
TreeSet(new PQsortByPriorityAndSubmittionTime());
+
+//Topologies that was attempted to be scheduled but wasn't successull
+private TreeSet attemptedQueue = new 
TreeSet(new PQsortByPriorityAndSubmittionTime());
--- End diff --

ok will correct the spelling but I will leave the name of the variable as 
it is for now


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064497#comment-15064497
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48057662
  
--- Diff: 
storm-core/src/jvm/backtype/storm/validation/ConfigValidation.java ---
@@ -489,17 +489,63 @@ public  void validateField(String name, Object o) {
 public static class PacemakerAuthTypeValidator extends Validator {
 @Override
 public void validateField(String name, Object o) {
-if(o == null) {
-throw new IllegalArgumentException( "Field " + name + " 
must be set.");
+if (o == null) {
+throw new IllegalArgumentException("Field " + name + " 
must be set.");
+}
+
+if (o instanceof String &&
+(((String) o).equals("NONE") ||
+((String) o).equals("DIGEST") ||
+((String) o).equals("KERBEROS"))) {
+return;
+}
+throw new IllegalArgumentException("Field " + name + " must be 
one of \"NONE\", \"DIGEST\", or \"KERBEROS\"");
+}
+}
--- End diff --

oh gotcha


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064495#comment-15064495
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48057606
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/priority/ISchedulingPriorityStrategy.java
 ---
@@ -16,22 +16,26 @@
  * limitations under the License.
  */
 
-package backtype.storm.scheduler.resource.strategies;
-
-import java.util.Collection;
-import java.util.Map;
+package backtype.storm.scheduler.resource.strategies.priority;
 
+import backtype.storm.scheduler.Cluster;
 import backtype.storm.scheduler.Topologies;
-import backtype.storm.scheduler.ExecutorDetails;
 import backtype.storm.scheduler.TopologyDetails;
-import backtype.storm.scheduler.WorkerSlot;
-import backtype.storm.scheduler.resource.RAS_Node;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
 
-/**
- * An interface to for implementing different scheduling strategies for 
the resource aware scheduling
- * In the future stategies will be pluggable
- */
-public interface IStrategy {
+import java.util.Map;
+
+public interface ISchedulingPriorityStrategy {
+
+/**
+ * initializes
+ */
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes);
--- End diff --

Ya but just like my comment in for the IEviction interface.  The 
DefaultSchedulingPriorityStrategy might not use those variables but someone 
might implement a strategy in the future that might.  And we don't want to have 
to come back and add those parameters in the future which would break backwards 
compatibility 


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064488#comment-15064488
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48057522
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/eviction/DefaultEvictionStrategy.java
 ---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.eviction;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Map;
+
+public class DefaultEvictionStrategy implements IEvictionStrategy {
+private static final Logger LOG = LoggerFactory
+.getLogger(DefaultEvictionStrategy.class);
+
+private Topologies topologies;
+private Cluster cluster;
+private Map userMap;
+private RAS_Nodes nodes;
+
+@Override
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes) {
+this.topologies = topologies;
+this.cluster = cluster;
+this.userMap = userMap;
+this.nodes = nodes;
+}
+
+@Override
+public boolean makeSpaceForTopo(TopologyDetails td) {
+LOG.debug("attempting to make space for topo {} from user {}", 
td.getName(), td.getTopologySubmitter());
+User submitter = this.userMap.get(td.getTopologySubmitter());
+if (submitter.getCPUResourceGuaranteed() == null || 
submitter.getMemoryResourceGuaranteed() == null) {
+return false;
+}
+double cpuNeeded = td.getTotalRequestedCpu() / 
submitter.getCPUResourceGuaranteed();
+double memoryNeeded = (td.getTotalRequestedMemOffHeap() + 
td.getTotalRequestedMemOnHeap()) / submitter.getMemoryResourceGuaranteed();
+
+User evictUser = 
this.findUserWithHighestAverageResourceUtilAboveGuarantee();
+//user has enough resource under his or her resource guarantee to 
schedule topology
+if ((1.0 - submitter.getCPUResourcePoolUtilization()) >= cpuNeeded 
&& (1.0 - submitter.getMemoryResourcePoolUtilization()) >= memoryNeeded) {
+if (evictUser != null) {
+TopologyDetails topologyEvict = 
evictUser.getRunningTopologyWithLowestPriority();
+evictTopology(topologyEvict);
+return true;
+}
+} else {
+if (evictUser != null) {
+if ((evictUser.getResourcePoolAverageUtilization() - 1.0) 
> (cpuNeeded + (submitter.getResourcePoolAverageUtilization() - 1.0))) {
+TopologyDetails topologyEvict = 
evictUser.getRunningTopologyWithLowestPriority();
+evictTopology(topologyEvict);
+return true;
+}
+}
+}
+//See if there is a lower priority topology that can be evicted 
from the current user
+for (TopologyDetails topo : submitter.getTopologiesRunning()) {
+//check to if there is a topology with a lower priority we can 
evict
+if (topo.getTopologyPriority() > td.getTopologyPriority()) {
+evictTopology(topo);
+return true;
+}
+}
+return false;
+}
+
+private void evictTopology(TopologyDetails topologyEvict) {
+Collection workersToEvict = 
this.cluster.getUsedSlotsByTopologyId(topologyEvict.getId())

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064433#comment-15064433
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48054747
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/eviction/DefaultEvictionStrategy.java
 ---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.eviction;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Map;
+
+public class DefaultEvictionStrategy implements IEvictionStrategy {
+private static final Logger LOG = LoggerFactory
+.getLogger(DefaultEvictionStrategy.class);
+
+private Topologies topologies;
+private Cluster cluster;
+private Map userMap;
+private RAS_Nodes nodes;
+
+@Override
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes) {
+this.topologies = topologies;
+this.cluster = cluster;
+this.userMap = userMap;
+this.nodes = nodes;
+}
+
+@Override
+public boolean makeSpaceForTopo(TopologyDetails td) {
+LOG.debug("attempting to make space for topo {} from user {}", 
td.getName(), td.getTopologySubmitter());
+User submitter = this.userMap.get(td.getTopologySubmitter());
+if (submitter.getCPUResourceGuaranteed() == null || 
submitter.getMemoryResourceGuaranteed() == null) {
+return false;
+}
+double cpuNeeded = td.getTotalRequestedCpu() / 
submitter.getCPUResourceGuaranteed();
+double memoryNeeded = (td.getTotalRequestedMemOffHeap() + 
td.getTotalRequestedMemOnHeap()) / submitter.getMemoryResourceGuaranteed();
+
+User evictUser = 
this.findUserWithHighestAverageResourceUtilAboveGuarantee();
+//user has enough resource under his or her resource guarantee to 
schedule topology
+if ((1.0 - submitter.getCPUResourcePoolUtilization()) >= cpuNeeded 
&& (1.0 - submitter.getMemoryResourcePoolUtilization()) >= memoryNeeded) {
+if (evictUser != null) {
+TopologyDetails topologyEvict = 
evictUser.getRunningTopologyWithLowestPriority();
+evictTopology(topologyEvict);
+return true;
+}
+} else {
+if (evictUser != null) {
+if ((evictUser.getResourcePoolAverageUtilization() - 1.0) 
> (cpuNeeded + (submitter.getResourcePoolAverageUtilization() - 1.0))) {
+TopologyDetails topologyEvict = 
evictUser.getRunningTopologyWithLowestPriority();
+evictTopology(topologyEvict);
+return true;
+}
+}
+}
+//See if there is a lower priority topology that can be evicted 
from the current user
+for (TopologyDetails topo : submitter.getTopologiesRunning()) {
+//check to if there is a topology with a lower priority we can 
evict
+if (topo.getTopologyPriority() > td.getTopologyPriority()) {
+evictTopology(topo);
+return true;
+}
+}
--- End diff --

will add


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
>  

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064383#comment-15064383
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48052644
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/SchedulingResult.java ---
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Map;
+
+/**
+ * This class serves as a mechanism to return results and messages from a 
scheduling strategy to the Resource Aware Scheduler
+ */
+public class SchedulingResult {
+
+//contains the result for the attempted scheduling
+private Map> 
schedulingResultMap = null;
+
+//status of scheduling the topology e.g. success or fail?
+private SchedulingStatus status = null;
+
+//arbitrary message to be returned when scheduling is done
+private String message = null;
+
+//error message returned is something went wrong
+private String errorMessage = null;
+
+private static final Logger LOG = 
LoggerFactory.getLogger(SchedulingResult.class);
+
+private SchedulingResult(SchedulingStatus status, Map> schedulingResultMap, String message, String 
errorMessage) {
+this.status = status;
+this.schedulingResultMap = schedulingResultMap;
+this.message = message;
+this.errorMessage = errorMessage;
+}
+
+public static SchedulingResult failure(SchedulingStatus status, String 
errorMessage) {
+return new SchedulingResult(status, null, null, errorMessage);
+}
+
+public static SchedulingResult success(Map> schedulingResultMap) {
+return SchedulingResult.successWithMsg(schedulingResultMap, null);
+}
+
+public static SchedulingResult successWithMsg(Map> schedulingResultMap, String message) {
+if (schedulingResultMap == null) {
+throw new IllegalStateException("Cannot declare scheduling 
success without providing a non null scheduling map!");
+}
+return new SchedulingResult(SchedulingStatus.SUCCESS, 
schedulingResultMap, message, null);
+}
+
+public SchedulingStatus getStatus() {
+return this.status;
+}
+
+public String getMessage() {
+return this.message;
+}
+
+public String getErrorMessage() {
+return this.errorMessage;
+}
--- End diff --

ok will add comment


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> I

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064381#comment-15064381
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48052613
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/ResourceAwareScheduler.java
 ---
@@ -28,94 +32,270 @@
 import backtype.storm.scheduler.Topologies;
 import backtype.storm.scheduler.TopologyDetails;
 import backtype.storm.scheduler.WorkerSlot;
-import backtype.storm.scheduler.resource.strategies.ResourceAwareStrategy;
 
+import java.util.Arrays;
 import java.util.Collection;
 import java.util.HashMap;
 import java.util.HashSet;
-import java.util.LinkedList;
 import java.util.Map;
 import java.util.Set;
 
 public class ResourceAwareScheduler implements IScheduler {
+
+private Map userMap;
+private Cluster cluster;
+private Topologies topologies;
+private RAS_Nodes nodes;
+
+private class SchedulingState {
+private Map userMap = new HashMap();
+private Cluster cluster;
+private Topologies topologies;
+private RAS_Nodes nodes;
+private Map conf = new Config();
+
+public SchedulingState(Map userMap, Cluster cluster, 
Topologies topologies, RAS_Nodes nodes, Map conf) {
+for (Map.Entry userMapEntry : 
userMap.entrySet()) {
+String userId = userMapEntry.getKey();
+User user = userMapEntry.getValue();
+this.userMap.put(userId, user.getCopy());
+}
+this.cluster = Cluster.getCopy(cluster);
+this.topologies = topologies.getCopy(topologies);
+this.nodes = new RAS_Nodes(this.cluster, this.topologies);
+this.conf.putAll(conf);
+}
+}
+
+@SuppressWarnings("rawtypes")
+private Map conf;
+
 private static final Logger LOG = LoggerFactory
 .getLogger(ResourceAwareScheduler.class);
-@SuppressWarnings("rawtypes")
-private Map _conf;
 
 @Override
 public void prepare(Map conf) {
-_conf = conf;
+this.conf = conf;
+
 }
 
 @Override
 public void schedule(Topologies topologies, Cluster cluster) {
 LOG.debug("\n\n\nRerunning ResourceAwareScheduler...");
+//initialize data structures
+this.initialize(topologies, cluster);
--- End diff --

will get rid of the "this" for all method calls


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064374#comment-15064374
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/921#issuecomment-165859285
  
> In the future, perhaps the concept of a worker slot will disappear.

I had the same thought.  Myself, I thought of slots as "logical resources" 
necessary only because we did not have another way of managing resources.

> Just for more clarification the eviction policy works independently of 
the number of slots. The eviction policy will be evaluated every time cannot be 
scheduled regardless the reason (i.e. run out of slots or run out of resources).

Great, sounds good.

> @d2r thank you so much for your detailed review! 

Welcome, and thanks for building this, it will be valuable.




> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064363#comment-15064363
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48052047
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/scheduling/DefaultResourceAwareStrategy.java
 ---
@@ -97,7 +101,7 @@ public ResourceAwareStrategy(Cluster cluster, Topologies 
topologies) {
 
 if (spouts.size() == 0) {
 LOG.error("Cannot find a Spout!");
-return null;
+return 
SchedulingResult.failure(SchedulingStatus.FAIL_INVALID_TOPOLOGY, "Cannot find a 
Spout!");
--- End diff --

sure thing


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064348#comment-15064348
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48051492
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestUser.java ---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import org.junit.Assert;
+import org.junit.Test;
--- End diff --

You are correct, my mistake.


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064345#comment-15064345
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48051405
  
--- Diff: storm-core/src/jvm/backtype/storm/scheduler/resource/User.java ---
@@ -0,0 +1,345 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.TopologyDetails;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeSet;
+
+public class User {
+private String userId;
+//Topologies yet to be scheduled sorted by priority for each user
+private TreeSet pendingQueue = new 
TreeSet(new PQsortByPriorityAndSubmittionTime());
+
+//Topologies yet to be scheduled sorted by priority for each user
+private TreeSet runningQueue = new 
TreeSet(new PQsortByPriorityAndSubmittionTime());
+
+//Topologies that was attempted to be scheduled but wasn't successull
+private TreeSet attemptedQueue = new 
TreeSet(new PQsortByPriorityAndSubmittionTime());
--- End diff --

`successful`

Also, we could rename to something like `failedAttemptsQueue` for clarity, 
but that is not a big deal.


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064340#comment-15064340
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48051291
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails topo1 = 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064324#comment-15064324
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48050801
  
--- Diff: 
storm-core/src/jvm/backtype/storm/validation/ConfigValidation.java ---
@@ -489,17 +489,63 @@ public  void validateField(String name, Object o) {
 public static class PacemakerAuthTypeValidator extends Validator {
 @Override
 public void validateField(String name, Object o) {
-if(o == null) {
-throw new IllegalArgumentException( "Field " + name + " 
must be set.");
+if (o == null) {
+throw new IllegalArgumentException("Field " + name + " 
must be set.");
+}
+
+if (o instanceof String &&
+(((String) o).equals("NONE") ||
+((String) o).equals("DIGEST") ||
+((String) o).equals("KERBEROS"))) {
+return;
+}
+throw new IllegalArgumentException("Field " + name + " must be 
one of \"NONE\", \"DIGEST\", or \"KERBEROS\"");
+}
+}
--- End diff --

OK, it picked up the space after the `if`: `if(` -> `if (`. I am fine with 
the changes as they are.


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064313#comment-15064313
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48050567
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/priority/ISchedulingPriorityStrategy.java
 ---
@@ -16,22 +16,26 @@
  * limitations under the License.
  */
 
-package backtype.storm.scheduler.resource.strategies;
-
-import java.util.Collection;
-import java.util.Map;
+package backtype.storm.scheduler.resource.strategies.priority;
 
+import backtype.storm.scheduler.Cluster;
 import backtype.storm.scheduler.Topologies;
-import backtype.storm.scheduler.ExecutorDetails;
 import backtype.storm.scheduler.TopologyDetails;
-import backtype.storm.scheduler.WorkerSlot;
-import backtype.storm.scheduler.resource.RAS_Node;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
 
-/**
- * An interface to for implementing different scheduling strategies for 
the resource aware scheduling
- * In the future stategies will be pluggable
- */
-public interface IStrategy {
+import java.util.Map;
+
+public interface ISchedulingPriorityStrategy {
+
+/**
+ * initializes
+ */
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes);
--- End diff --

I am asking because the DefaultSchedulingPriorityStrategy does not use 
`topologies` or `nodes`. It looks like they are not needed for anything.


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064308#comment-15064308
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48050274
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/eviction/DefaultEvictionStrategy.java
 ---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.eviction;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Map;
+
+public class DefaultEvictionStrategy implements IEvictionStrategy {
+private static final Logger LOG = LoggerFactory
+.getLogger(DefaultEvictionStrategy.class);
+
+private Topologies topologies;
+private Cluster cluster;
+private Map userMap;
+private RAS_Nodes nodes;
+
+@Override
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes) {
+this.topologies = topologies;
+this.cluster = cluster;
+this.userMap = userMap;
+this.nodes = nodes;
+}
+
+@Override
+public boolean makeSpaceForTopo(TopologyDetails td) {
+LOG.debug("attempting to make space for topo {} from user {}", 
td.getName(), td.getTopologySubmitter());
+User submitter = this.userMap.get(td.getTopologySubmitter());
+if (submitter.getCPUResourceGuaranteed() == null || 
submitter.getMemoryResourceGuaranteed() == null) {
+return false;
+}
+double cpuNeeded = td.getTotalRequestedCpu() / 
submitter.getCPUResourceGuaranteed();
+double memoryNeeded = (td.getTotalRequestedMemOffHeap() + 
td.getTotalRequestedMemOnHeap()) / submitter.getMemoryResourceGuaranteed();
+
+User evictUser = 
this.findUserWithHighestAverageResourceUtilAboveGuarantee();
+//user has enough resource under his or her resource guarantee to 
schedule topology
+if ((1.0 - submitter.getCPUResourcePoolUtilization()) >= cpuNeeded 
&& (1.0 - submitter.getMemoryResourcePoolUtilization()) >= memoryNeeded) {
+if (evictUser != null) {
+TopologyDetails topologyEvict = 
evictUser.getRunningTopologyWithLowestPriority();
+evictTopology(topologyEvict);
+return true;
+}
+} else {
+if (evictUser != null) {
+if ((evictUser.getResourcePoolAverageUtilization() - 1.0) 
> (cpuNeeded + (submitter.getResourcePoolAverageUtilization() - 1.0))) {
+TopologyDetails topologyEvict = 
evictUser.getRunningTopologyWithLowestPriority();
+evictTopology(topologyEvict);
+return true;
+}
+}
+}
+//See if there is a lower priority topology that can be evicted 
from the current user
+for (TopologyDetails topo : submitter.getTopologiesRunning()) {
+//check to if there is a topology with a lower priority we can 
evict
+if (topo.getTopologyPriority() > td.getTopologyPriority()) {
+evictTopology(topo);
+return true;
+}
+}
+return false;
+}
+
+private void evictTopology(TopologyDetails topologyEvict) {
+Collection workersToEvict = 
this.cluster.getUsedSlotsByTopologyId(topologyEvict.getId())

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064304#comment-15064304
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48049892
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/eviction/DefaultEvictionStrategy.java
 ---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.eviction;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Map;
+
+public class DefaultEvictionStrategy implements IEvictionStrategy {
+private static final Logger LOG = LoggerFactory
+.getLogger(DefaultEvictionStrategy.class);
+
+private Topologies topologies;
+private Cluster cluster;
+private Map userMap;
+private RAS_Nodes nodes;
+
+@Override
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes) {
+this.topologies = topologies;
+this.cluster = cluster;
+this.userMap = userMap;
+this.nodes = nodes;
+}
+
+@Override
+public boolean makeSpaceForTopo(TopologyDetails td) {
+LOG.debug("attempting to make space for topo {} from user {}", 
td.getName(), td.getTopologySubmitter());
+User submitter = this.userMap.get(td.getTopologySubmitter());
+if (submitter.getCPUResourceGuaranteed() == null || 
submitter.getMemoryResourceGuaranteed() == null) {
+return false;
+}
+double cpuNeeded = td.getTotalRequestedCpu() / 
submitter.getCPUResourceGuaranteed();
+double memoryNeeded = (td.getTotalRequestedMemOffHeap() + 
td.getTotalRequestedMemOnHeap()) / submitter.getMemoryResourceGuaranteed();
+
+User evictUser = 
this.findUserWithHighestAverageResourceUtilAboveGuarantee();
+//user has enough resource under his or her resource guarantee to 
schedule topology
+if ((1.0 - submitter.getCPUResourcePoolUtilization()) >= cpuNeeded 
&& (1.0 - submitter.getMemoryResourcePoolUtilization()) >= memoryNeeded) {
+if (evictUser != null) {
+TopologyDetails topologyEvict = 
evictUser.getRunningTopologyWithLowestPriority();
+evictTopology(topologyEvict);
+return true;
+}
+} else {
+if (evictUser != null) {
+if ((evictUser.getResourcePoolAverageUtilization() - 1.0) 
> (cpuNeeded + (submitter.getResourcePoolAverageUtilization() - 1.0))) {
+TopologyDetails topologyEvict = 
evictUser.getRunningTopologyWithLowestPriority();
+evictTopology(topologyEvict);
+return true;
+}
+}
+}
+//See if there is a lower priority topology that can be evicted 
from the current user
+for (TopologyDetails topo : submitter.getTopologiesRunning()) {
+//check to if there is a topology with a lower priority we can 
evict
+if (topo.getTopologyPriority() > td.getTopologyPriority()) {
+evictTopology(topo);
+return true;
+}
+}
--- End diff --

Yes I did not think of that.  Even so, once we pass this point, we lose the 
information about the reason it was considered for eviction.  What if we add a 
DEBUG log message to the effe

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064299#comment-15064299
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48049605
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/eviction/IEvictionStrategy.java
 ---
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.eviction;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+
+import java.util.Map;
+
+public interface IEvictionStrategy {
+
+/**
+ * Initialization
+ */
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes);
--- End diff --

Good answer. Agreed.


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064298#comment-15064298
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48049554
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/SchedulingResult.java ---
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Map;
+
+/**
+ * This class serves as a mechanism to return results and messages from a 
scheduling strategy to the Resource Aware Scheduler
+ */
+public class SchedulingResult {
+
+//contains the result for the attempted scheduling
+private Map> 
schedulingResultMap = null;
+
+//status of scheduling the topology e.g. success or fail?
+private SchedulingStatus status = null;
+
+//arbitrary message to be returned when scheduling is done
+private String message = null;
+
+//error message returned is something went wrong
+private String errorMessage = null;
+
+private static final Logger LOG = 
LoggerFactory.getLogger(SchedulingResult.class);
+
+private SchedulingResult(SchedulingStatus status, Map> schedulingResultMap, String message, String 
errorMessage) {
+this.status = status;
+this.schedulingResultMap = schedulingResultMap;
+this.message = message;
+this.errorMessage = errorMessage;
+}
+
+public static SchedulingResult failure(SchedulingStatus status, String 
errorMessage) {
+return new SchedulingResult(status, null, null, errorMessage);
+}
+
+public static SchedulingResult success(Map> schedulingResultMap) {
+return SchedulingResult.successWithMsg(schedulingResultMap, null);
+}
+
+public static SchedulingResult successWithMsg(Map> schedulingResultMap, String message) {
+if (schedulingResultMap == null) {
+throw new IllegalStateException("Cannot declare scheduling 
success without providing a non null scheduling map!");
+}
+return new SchedulingResult(SchedulingStatus.SUCCESS, 
schedulingResultMap, message, null);
+}
+
+public SchedulingStatus getStatus() {
+return this.status;
+}
+
+public String getMessage() {
+return this.message;
+}
+
+public String getErrorMessage() {
+return this.errorMessage;
+}
--- End diff --

Ah, I see. Let's comment that the `message` is really just intended to be a 
short summary of what happened.  It is used in `toString` and in tests. That 
should be enough to explain if someone else is wondering why they are both 
there. 


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their to

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064286#comment-15064286
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48049016
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/ResourceAwareScheduler.java
 ---
@@ -128,56 +308,121 @@ private void updateSupervisorsResources(Cluster 
cluster, Topologies topologies)
 cluster.setSupervisorsResourcesMap(supervisors_resources);
 }
 
-private Map getUserConf() {
-Map ret = new HashMap();
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB));
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB));
-return ret;
+public User getUser(String user) {
+return this.userMap.get(user);
+}
+
+public Map getUserMap() {
+return this.userMap;
 }
 
 /**
- * print scheduling for debug purposes
- * @param cluster
+ * Intialize scheduling and running queues
+ *
  * @param topologies
+ * @param cluster
  */
-public String printScheduling(Cluster cluster, Topologies topologies) {
-StringBuilder str = new StringBuilder();
-Map>>> schedulingMap = new HashMap>>>();
-for (TopologyDetails topo : topologies.getTopologies()) {
-if (cluster.getAssignmentById(topo.getId()) != null) {
-for (Map.Entry entry : 
cluster.getAssignmentById(topo.getId()).getExecutorToSlot().entrySet()) {
-WorkerSlot slot = entry.getValue();
-String nodeId = slot.getNodeId();
-ExecutorDetails exec = entry.getKey();
-if (schedulingMap.containsKey(nodeId) == false) {
-schedulingMap.put(nodeId, new HashMap>>());
-}
-if 
(schedulingMap.get(nodeId).containsKey(topo.getId()) == false) {
-schedulingMap.get(nodeId).put(topo.getId(), new 
HashMap>());
-}
-if 
(schedulingMap.get(nodeId).get(topo.getId()).containsKey(slot) == false) {
-
schedulingMap.get(nodeId).get(topo.getId()).put(slot, new 
LinkedList());
-}
-
schedulingMap.get(nodeId).get(topo.getId()).get(slot).add(exec);
+private void initUsers(Topologies topologies, Cluster cluster) {
+this.userMap = new HashMap();
+Map> userResourcePools = 
this.getUserResourcePools();
+LOG.debug("userResourcePools: {}", userResourcePools);
+
+for (TopologyDetails td : topologies.getTopologies()) {
+//Get user that submitted topology.  If topology submitter is 
null or empty string, the topologySubmitter
+//will be set to anonymous
+String topologySubmitter = td.getTopologySubmitter();
+//additional safety check to make sure that topologySubmitter 
is going to be a valid value
+if (topologySubmitter == null || topologySubmitter.equals("")) 
{
+LOG.error("Cannot determine user for topology {}.  Will 
skip scheduling this topology", td.getName());
+continue;
+}
+if (!this.userMap.containsKey(topologySubmitter)) {
+this.userMap.put(topologySubmitter, new 
User(topologySubmitter, userResourcePools.get(topologySubmitter)));
+}
+if (cluster.getUnassignedExecutors(td).size() > 0) {
+LOG.debug("adding td: {} to pending queue", td.getName());
+
this.userMap.get(topologySubmitter).addTopologyToPendingQueue(td);
+} else {
+LOG.debug("adding td: {} to running queue with existing 
status: {}", td.getName(), cluster.getStatusMap().get(td.getId()));
+
this.userMap.get(topologySubmitter).addTopologyToRunningQueue(td);
+if (cluster.getStatusMap().get(td.getId()) == null || 
cluster.getStatusMap().get(td.getId()).equals("")) {
+cluster.setStatus(td.getId(), "Fully Scheduled");
 }
 }
 }
+}
 
-for (Map.Entry>>> entry : schedulingMap.entrySet()) {
-if (cluster.getSupervisorById(entry.getKey()) != null) {
-str.append("/** Node: 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064285#comment-15064285
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48048970
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/ResourceAwareScheduler.java
 ---
@@ -128,56 +308,121 @@ private void updateSupervisorsResources(Cluster 
cluster, Topologies topologies)
 cluster.setSupervisorsResourcesMap(supervisors_resources);
 }
 
-private Map getUserConf() {
-Map ret = new HashMap();
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB));
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB));
-return ret;
+public User getUser(String user) {
+return this.userMap.get(user);
+}
+
+public Map getUserMap() {
+return this.userMap;
 }
 
 /**
- * print scheduling for debug purposes
- * @param cluster
+ * Intialize scheduling and running queues
+ *
  * @param topologies
+ * @param cluster
  */
-public String printScheduling(Cluster cluster, Topologies topologies) {
-StringBuilder str = new StringBuilder();
-Map>>> schedulingMap = new HashMap>>>();
-for (TopologyDetails topo : topologies.getTopologies()) {
-if (cluster.getAssignmentById(topo.getId()) != null) {
-for (Map.Entry entry : 
cluster.getAssignmentById(topo.getId()).getExecutorToSlot().entrySet()) {
-WorkerSlot slot = entry.getValue();
-String nodeId = slot.getNodeId();
-ExecutorDetails exec = entry.getKey();
-if (schedulingMap.containsKey(nodeId) == false) {
-schedulingMap.put(nodeId, new HashMap>>());
-}
-if 
(schedulingMap.get(nodeId).containsKey(topo.getId()) == false) {
-schedulingMap.get(nodeId).put(topo.getId(), new 
HashMap>());
-}
-if 
(schedulingMap.get(nodeId).get(topo.getId()).containsKey(slot) == false) {
-
schedulingMap.get(nodeId).get(topo.getId()).put(slot, new 
LinkedList());
-}
-
schedulingMap.get(nodeId).get(topo.getId()).get(slot).add(exec);
+private void initUsers(Topologies topologies, Cluster cluster) {
+this.userMap = new HashMap();
+Map> userResourcePools = 
this.getUserResourcePools();
+LOG.debug("userResourcePools: {}", userResourcePools);
+
+for (TopologyDetails td : topologies.getTopologies()) {
+//Get user that submitted topology.  If topology submitter is 
null or empty string, the topologySubmitter
+//will be set to anonymous
+String topologySubmitter = td.getTopologySubmitter();
+//additional safety check to make sure that topologySubmitter 
is going to be a valid value
+if (topologySubmitter == null || topologySubmitter.equals("")) 
{
+LOG.error("Cannot determine user for topology {}.  Will 
skip scheduling this topology", td.getName());
+continue;
+}
+if (!this.userMap.containsKey(topologySubmitter)) {
+this.userMap.put(topologySubmitter, new 
User(topologySubmitter, userResourcePools.get(topologySubmitter)));
+}
+if (cluster.getUnassignedExecutors(td).size() > 0) {
+LOG.debug("adding td: {} to pending queue", td.getName());
+
this.userMap.get(topologySubmitter).addTopologyToPendingQueue(td);
+} else {
+LOG.debug("adding td: {} to running queue with existing 
status: {}", td.getName(), cluster.getStatusMap().get(td.getId()));
+
this.userMap.get(topologySubmitter).addTopologyToRunningQueue(td);
+if (cluster.getStatusMap().get(td.getId()) == null || 
cluster.getStatusMap().get(td.getId()).equals("")) {
+cluster.setStatus(td.getId(), "Fully Scheduled");
 }
 }
 }
+}
 
-for (Map.Entry>>> entry : schedulingMap.entrySet()) {
-if (cluster.getSupervisorById(entry.getKey()) != null) {
-str.append("/** Node: 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064284#comment-15064284
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48048927
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/ResourceAwareScheduler.java
 ---
@@ -128,56 +308,121 @@ private void updateSupervisorsResources(Cluster 
cluster, Topologies topologies)
 cluster.setSupervisorsResourcesMap(supervisors_resources);
 }
 
-private Map getUserConf() {
-Map ret = new HashMap();
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB));
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB));
-return ret;
+public User getUser(String user) {
+return this.userMap.get(user);
+}
+
+public Map getUserMap() {
+return this.userMap;
 }
 
 /**
- * print scheduling for debug purposes
- * @param cluster
+ * Intialize scheduling and running queues
+ *
  * @param topologies
+ * @param cluster
  */
-public String printScheduling(Cluster cluster, Topologies topologies) {
-StringBuilder str = new StringBuilder();
-Map>>> schedulingMap = new HashMap>>>();
-for (TopologyDetails topo : topologies.getTopologies()) {
-if (cluster.getAssignmentById(topo.getId()) != null) {
-for (Map.Entry entry : 
cluster.getAssignmentById(topo.getId()).getExecutorToSlot().entrySet()) {
-WorkerSlot slot = entry.getValue();
-String nodeId = slot.getNodeId();
-ExecutorDetails exec = entry.getKey();
-if (schedulingMap.containsKey(nodeId) == false) {
-schedulingMap.put(nodeId, new HashMap>>());
-}
-if 
(schedulingMap.get(nodeId).containsKey(topo.getId()) == false) {
-schedulingMap.get(nodeId).put(topo.getId(), new 
HashMap>());
-}
-if 
(schedulingMap.get(nodeId).get(topo.getId()).containsKey(slot) == false) {
-
schedulingMap.get(nodeId).get(topo.getId()).put(slot, new 
LinkedList());
-}
-
schedulingMap.get(nodeId).get(topo.getId()).get(slot).add(exec);
+private void initUsers(Topologies topologies, Cluster cluster) {
+this.userMap = new HashMap();
+Map> userResourcePools = 
this.getUserResourcePools();
+LOG.debug("userResourcePools: {}", userResourcePools);
+
+for (TopologyDetails td : topologies.getTopologies()) {
+//Get user that submitted topology.  If topology submitter is 
null or empty string, the topologySubmitter
+//will be set to anonymous
+String topologySubmitter = td.getTopologySubmitter();
+//additional safety check to make sure that topologySubmitter 
is going to be a valid value
+if (topologySubmitter == null || topologySubmitter.equals("")) 
{
+LOG.error("Cannot determine user for topology {}.  Will 
skip scheduling this topology", td.getName());
+continue;
+}
+if (!this.userMap.containsKey(topologySubmitter)) {
+this.userMap.put(topologySubmitter, new 
User(topologySubmitter, userResourcePools.get(topologySubmitter)));
+}
+if (cluster.getUnassignedExecutors(td).size() > 0) {
+LOG.debug("adding td: {} to pending queue", td.getName());
+
this.userMap.get(topologySubmitter).addTopologyToPendingQueue(td);
+} else {
+LOG.debug("adding td: {} to running queue with existing 
status: {}", td.getName(), cluster.getStatusMap().get(td.getId()));
+
this.userMap.get(topologySubmitter).addTopologyToRunningQueue(td);
+if (cluster.getStatusMap().get(td.getId()) == null || 
cluster.getStatusMap().get(td.getId()).equals("")) {
+cluster.setStatus(td.getId(), "Fully Scheduled");
 }
 }
 }
+}
 
-for (Map.Entry>>> entry : schedulingMap.entrySet()) {
-if (cluster.getSupervisorById(entry.getKey()) != null) {
-str.append("/** Node: 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064269#comment-15064269
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48048580
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/ResourceAwareScheduler.java
 ---
@@ -28,94 +32,270 @@
 import backtype.storm.scheduler.Topologies;
 import backtype.storm.scheduler.TopologyDetails;
 import backtype.storm.scheduler.WorkerSlot;
-import backtype.storm.scheduler.resource.strategies.ResourceAwareStrategy;
 
+import java.util.Arrays;
 import java.util.Collection;
 import java.util.HashMap;
 import java.util.HashSet;
-import java.util.LinkedList;
 import java.util.Map;
 import java.util.Set;
 
 public class ResourceAwareScheduler implements IScheduler {
+
+private Map userMap;
+private Cluster cluster;
+private Topologies topologies;
+private RAS_Nodes nodes;
+
+private class SchedulingState {
+private Map userMap = new HashMap();
+private Cluster cluster;
+private Topologies topologies;
+private RAS_Nodes nodes;
+private Map conf = new Config();
+
+public SchedulingState(Map userMap, Cluster cluster, 
Topologies topologies, RAS_Nodes nodes, Map conf) {
+for (Map.Entry userMapEntry : 
userMap.entrySet()) {
+String userId = userMapEntry.getKey();
+User user = userMapEntry.getValue();
+this.userMap.put(userId, user.getCopy());
+}
+this.cluster = Cluster.getCopy(cluster);
+this.topologies = topologies.getCopy(topologies);
+this.nodes = new RAS_Nodes(this.cluster, this.topologies);
+this.conf.putAll(conf);
+}
+}
+
+@SuppressWarnings("rawtypes")
+private Map conf;
+
 private static final Logger LOG = LoggerFactory
 .getLogger(ResourceAwareScheduler.class);
-@SuppressWarnings("rawtypes")
-private Map _conf;
 
 @Override
 public void prepare(Map conf) {
-_conf = conf;
+this.conf = conf;
+
 }
 
 @Override
 public void schedule(Topologies topologies, Cluster cluster) {
 LOG.debug("\n\n\nRerunning ResourceAwareScheduler...");
+//initialize data structures
+this.initialize(topologies, cluster);
+//logs everything that is currently scheduled and the location at 
which they are scheduled
+LOG.info("Cluster scheduling:\n{}", 
ResourceUtils.printScheduling(cluster, topologies));
+//logs the resources available/used for every node
+LOG.info("Nodes:\n{}", this.nodes);
+//logs the detailed info about each user
+for (User user : this.getUserMap().values()) {
+LOG.info(user.getDetailedInfo());
+}
 
-ResourceAwareStrategy RAStrategy = new 
ResourceAwareStrategy(cluster, topologies);
-LOG.debug(printScheduling(cluster, topologies));
+ISchedulingPriorityStrategy schedulingPrioritystrategy = null;
+while (true) {
 
-for (TopologyDetails td : topologies.getTopologies()) {
-String topId = td.getId();
-Map> 
schedulerAssignmentMap;
-if (cluster.getUnassignedExecutors(td).size() > 0) {
-LOG.debug("/Scheduling topology {} /", 
topId);
-
-schedulerAssignmentMap = RAStrategy.schedule(td);
-
-double requestedMemOnHeap = 
td.getTotalRequestedMemOnHeap();
-double requestedMemOffHeap = 
td.getTotalRequestedMemOffHeap();
-double requestedCpu = td.getTotalRequestedCpu();
-double assignedMemOnHeap = 0.0;
-double assignedMemOffHeap = 0.0;
-double assignedCpu = 0.0;
-
-if (schedulerAssignmentMap != null) {
-try {
-Set nodesUsed = new HashSet();
-int assignedWorkers = 
schedulerAssignmentMap.keySet().size();
-for (Map.Entry> workerToTasksEntry : 
schedulerAssignmentMap.entrySet()) {
-WorkerSlot targetSlot = 
workerToTasksEntry.getKey();
-Collection 
execsNeedScheduling = workerToTasksEntry.getValue();
-RAS_Node targetNode = 
RAStrategy.idToNode(targetSlot.getNodeId());
-  

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064251#comment-15064251
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48047678
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/ResourceAwareScheduler.java
 ---
@@ -28,94 +32,270 @@
 import backtype.storm.scheduler.Topologies;
 import backtype.storm.scheduler.TopologyDetails;
 import backtype.storm.scheduler.WorkerSlot;
-import backtype.storm.scheduler.resource.strategies.ResourceAwareStrategy;
 
+import java.util.Arrays;
 import java.util.Collection;
 import java.util.HashMap;
 import java.util.HashSet;
-import java.util.LinkedList;
 import java.util.Map;
 import java.util.Set;
 
 public class ResourceAwareScheduler implements IScheduler {
+
+private Map userMap;
+private Cluster cluster;
+private Topologies topologies;
+private RAS_Nodes nodes;
+
+private class SchedulingState {
+private Map userMap = new HashMap();
+private Cluster cluster;
+private Topologies topologies;
+private RAS_Nodes nodes;
+private Map conf = new Config();
+
+public SchedulingState(Map userMap, Cluster cluster, 
Topologies topologies, RAS_Nodes nodes, Map conf) {
+for (Map.Entry userMapEntry : 
userMap.entrySet()) {
+String userId = userMapEntry.getKey();
+User user = userMapEntry.getValue();
+this.userMap.put(userId, user.getCopy());
+}
+this.cluster = Cluster.getCopy(cluster);
+this.topologies = topologies.getCopy(topologies);
+this.nodes = new RAS_Nodes(this.cluster, this.topologies);
+this.conf.putAll(conf);
+}
+}
+
+@SuppressWarnings("rawtypes")
+private Map conf;
+
 private static final Logger LOG = LoggerFactory
 .getLogger(ResourceAwareScheduler.class);
-@SuppressWarnings("rawtypes")
-private Map _conf;
 
 @Override
 public void prepare(Map conf) {
-_conf = conf;
+this.conf = conf;
+
 }
 
 @Override
 public void schedule(Topologies topologies, Cluster cluster) {
 LOG.debug("\n\n\nRerunning ResourceAwareScheduler...");
+//initialize data structures
+this.initialize(topologies, cluster);
--- End diff --

The [version on 
master](https://github.com/apache/storm/blob/418792f84c95962a5f2e8d8f98aab73e8138c31a/storm-core/src/jvm/backtype/storm/scheduler/resource/ResourceAwareScheduler.java)
 right now does not use `this.` anywhere.  It uses the `_` prefix for member 
variables. This pull request changes the `_` prefix and also adds `this.` to 
method invocations, and both are probably not necessary.

In general we do not need to make changes solely for notation, but we 
definitely should not add `this.` for method invocations when they were not 
there before.


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology w

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064231#comment-15064231
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48046953
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/scheduling/DefaultResourceAwareStrategy.java
 ---
@@ -97,7 +101,7 @@ public ResourceAwareStrategy(Cluster cluster, Topologies 
topologies) {
 
 if (spouts.size() == 0) {
 LOG.error("Cannot find a Spout!");
-return null;
+return 
SchedulingResult.failure(SchedulingStatus.FAIL_INVALID_TOPOLOGY, "Cannot find a 
Spout!");
--- End diff --

Yeah, the DefaultTopologyValidator we have [does 
nothing](https://github.com/apache/storm/blob/418792f84c95962a5f2e8d8f98aab73e8138c31a/storm-core/src/jvm/backtype/storm/nimbus/DefaultTopologyValidator.java#L28-L30).
  I think it might be good to have a new Jira that adds a check to the 
DefaultValidator that at least one spout exists, and also removes code from 
this scheduler that repeats the same check.


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064221#comment-15064221
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48046580
  
--- Diff: docs/documentation/Resource_Aware_Scheduler_overview.md ---
@@ -0,0 +1,222 @@
+# Introduction
+
+The purpose of this document is to provide a description of the Resource 
Aware Scheduler for the Storm distributed real-time computation system.  This 
document will provide you with both a high level description of the resource 
aware scheduler in Storm
+
+## Using Resource Aware Scheduler
+
+The user can switch to using the Resource Aware Scheduler by setting the 
following in *conf/storm.yaml*
+
+storm.scheduler: 
“backtype.storm.scheduler.resource.ResourceAwareScheduler”
+
+
+## API Overview
+
+For a Storm Topology, the user can now specify the amount of resources a 
topology component (i.e. Spout or Bolt) is required to run a single instance of 
the component.  The user can specify the resource requirement for a topology 
component by using the following API calls.
+
+### Setting Memory Requirement
+
+API to set component memory requirement:
+
+public T setMemoryLoad(Number onHeap, Number offHeap)
+
+Parameters:
+* Number onHeap – The amount of on heap memory an instance of this 
component will consume in megabytes
+* Number OffHeap – The amount of off heap memory an instance of this 
component will consume in megabytes
+
+The user also have to option to just specify the on heap memory 
requirement if the component does not have an off heap memory need.
+
+public T setMemoryLoad(Number onHeap)
+
+Parameters:
+* Number onHeap – The amount of on heap memory an instance of this 
component will consume
+
+If no value is provided for offHeap, 0.0 will be used. If no value is 
provided for onHeap, or if the API is never called for a component, the default 
value will be used.
+
+Example of Usage:
+
+SpoutDeclarer s1 = builder.setSpout("word", new TestWordSpout(), 10);
+s1.setMemoryLoad(1024.0, 512.0);
+builder.setBolt("exclaim1", new ExclamationBolt(), 3)
+.shuffleGrouping("word").setMemoryLoad(512.0);
+
+The entire memory requested for this topology is 16.5 GB. That is from 10 
spouts with 1GB on heap memory and 0.5 GB off heap memory each and 3 bolts with 
0.5 GB on heap memory each.
+
+### Setting CPU Requirement
+
+
+API to set component CPU requirement:
+
+public T setCPULoad(Double amount)
+
+Parameters:
+* Number amount – The amount of on CPU an instance of this component will 
consume.
+
+Currently, the amount of CPU resources a component requires or is 
available on a node is represented by a point system. CPU usage is a difficult 
concept to define. Different CPU architectures perform differently depending on 
the task at hand. They are so complex that expressing all of that in a single 
precise portable number is impossible. Instead we take a convention over 
configuration approach and are primarily concerned with rough level of CPU 
usage while still providing the possibility to specify amounts more fine 
grained.
+
+By convention a CPU core typically will get 100 points. If you feel that 
your processors are more or less powerful you can adjust this accordingly. 
Heavy tasks that are CPU bound will get 100 points, as they can consume an 
entire core. Medium tasks should get 50, light tasks 25, and tiny tasks 10. In 
some cases you have a task that spawns other threads to help with processing. 
These tasks may need to go above 100 points to express the amount of CPU they 
are using. If these conventions are followed the common case for a single 
threaded task the reported Capacity * 100 should be the number of CPU points 
that the task needs.
+
+Example of Usage:
+
+SpoutDeclarer s1 = builder.setSpout("word", new TestWordSpout(), 10);
+s1.setCPULoad(15.0);
+builder.setBolt("exclaim1", new ExclamationBolt(), 3)
+.shuffleGrouping("word").setCPULoad(10.0);
+
+###Limiting the Heap Size per Worker (JVM) Process
+
+
+public void setTopologyWorkerMaxHeapSize(Number size)
+
+Parameters:
+* Number size – The memory limit a worker process will be allocated in 
megabytes
+
+The user can limit the amount of memory resources the resource aware 
scheduler that is allocated to a single worker on a per topology basis by using 
the above API.  This API is in place so that the users can spread executors to 
multiple workers.  However, spreading workers to mu

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064220#comment-15064220
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48046502
  
--- Diff: docs/documentation/Resource_Aware_Scheduler_overview.md ---
@@ -0,0 +1,227 @@
+# Introduction
+
+The purpose of this document is to provide a description of the Resource 
Aware Scheduler for the Storm distributed real-time computation system.  This 
document will provide you with both a high level description of the resource 
aware scheduler in Storm
+
+## Using Resource Aware Scheduler
+
+The user can switch to using the Resource Aware Scheduler by setting the 
following in *conf/storm.yaml*
+
+storm.scheduler: 
“backtype.storm.scheduler.resource.ResourceAwareScheduler”
+
+
+## API Overview
+
+For a Storm Topology, the user can now specify the amount of resources a 
topology component (i.e. Spout or Bolt) is required to run a single instance of 
the component.  The user can specify the resource requirement for a topology 
component by using the following API calls.
+
+### Setting Memory Requirement
+
+API to set component memory requirement:
+
+public T setMemoryLoad(Number onHeap, Number offHeap)
+
+Parameters:
+* Number onHeap – The amount of on heap memory an instance of this 
component will consume in megabytes
+* Number offHeap – The amount of off heap memory an instance of this 
component will consume in megabytes
+
+The user also has to option to just specify the on heap memory requirement 
if the component does not have an off heap memory need.
+
+public T setMemoryLoad(Number onHeap)
+
+Parameters:
+* Number onHeap – The amount of on heap memory an instance of this 
component will consume
+
+If no value is provided for offHeap, 0.0 will be used. If no value is 
provided for onHeap, or if the API is never called for a component, the default 
value will be used.
+
+Example of Usage:
+
+SpoutDeclarer s1 = builder.setSpout("word", new TestWordSpout(), 10);
+s1.setMemoryLoad(1024.0, 512.0);
+builder.setBolt("exclaim1", new ExclamationBolt(), 3)
+.shuffleGrouping("word").setMemoryLoad(512.0);
+
+The entire memory requested for this topology is 16.5 GB. That is from 10 
spouts with 1GB on heap memory and 0.5 GB off heap memory each and 3 bolts with 
0.5 GB on heap memory each.
+
+### Setting CPU Requirement
+
+
+API to set component CPU requirement:
+
+public T setCPULoad(Double amount)
+
+Parameters:
+* Number amount – The amount of on CPU an instance of this component will 
consume.
--- End diff --

Makes sense. If we rebalance to reduce parallelism, the memory resource 
requirement will be unchanged.  But for CPU it is not so clear.


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  W

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064192#comment-15064192
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/921#issuecomment-165837032
  
Just for more clarification the eviction policy works independently of the 
number of slots.  The eviction policy will be evaluated every time cannot be 
scheduled regardless the reason (i.e. run out of slots or run out of 
resources).  The default eviction policy will try to evict as many things as 
possible (within policy limits) to try to make space for a topology that was 
unsuccessfully scheduled


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063643#comment-15063643
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/921#issuecomment-165703496
  
@d2r thank you so much for your detailed review! Few people are will to 
take this much time and effort to review someone else's code.  Big thanks 
again! much appreciated!


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063640#comment-15063640
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on the pull request:

https://github.com/apache/storm/pull/921#issuecomment-165702815
  
If there no slots left on a node, the DefaultResourceAwareStrategy will not 
schedule on that node.  The number of slots should not effect the default 
eviction/priority strategy currently implemented. We could implement strategies 
that takes into account slots but I don't think that is necessary.  In the 
future, perhaps the concept of a worker slot will disappear.  Workers can be 
scheduled on nodes as long as there is free resources.  The size of a worker 
could be limited.  However, we should be careful and not spawn too many jvm 
process thus perhaps there should be an upper limit to the amount of jvm 
process a node can have


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063637#comment-15063637
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48000453
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/ResourceAwareScheduler.java
 ---
@@ -128,56 +308,121 @@ private void updateSupervisorsResources(Cluster 
cluster, Topologies topologies)
 cluster.setSupervisorsResourcesMap(supervisors_resources);
 }
 
-private Map getUserConf() {
-Map ret = new HashMap();
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB));
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB));
-return ret;
+public User getUser(String user) {
+return this.userMap.get(user);
+}
+
+public Map getUserMap() {
+return this.userMap;
 }
 
 /**
- * print scheduling for debug purposes
- * @param cluster
+ * Intialize scheduling and running queues
+ *
  * @param topologies
+ * @param cluster
  */
-public String printScheduling(Cluster cluster, Topologies topologies) {
-StringBuilder str = new StringBuilder();
-Map>>> schedulingMap = new HashMap>>>();
-for (TopologyDetails topo : topologies.getTopologies()) {
-if (cluster.getAssignmentById(topo.getId()) != null) {
-for (Map.Entry entry : 
cluster.getAssignmentById(topo.getId()).getExecutorToSlot().entrySet()) {
-WorkerSlot slot = entry.getValue();
-String nodeId = slot.getNodeId();
-ExecutorDetails exec = entry.getKey();
-if (schedulingMap.containsKey(nodeId) == false) {
-schedulingMap.put(nodeId, new HashMap>>());
-}
-if 
(schedulingMap.get(nodeId).containsKey(topo.getId()) == false) {
-schedulingMap.get(nodeId).put(topo.getId(), new 
HashMap>());
-}
-if 
(schedulingMap.get(nodeId).get(topo.getId()).containsKey(slot) == false) {
-
schedulingMap.get(nodeId).get(topo.getId()).put(slot, new 
LinkedList());
-}
-
schedulingMap.get(nodeId).get(topo.getId()).get(slot).add(exec);
+private void initUsers(Topologies topologies, Cluster cluster) {
+this.userMap = new HashMap();
+Map> userResourcePools = 
this.getUserResourcePools();
+LOG.debug("userResourcePools: {}", userResourcePools);
+
+for (TopologyDetails td : topologies.getTopologies()) {
+//Get user that submitted topology.  If topology submitter is 
null or empty string, the topologySubmitter
+//will be set to anonymous
+String topologySubmitter = td.getTopologySubmitter();
+//additional safety check to make sure that topologySubmitter 
is going to be a valid value
+if (topologySubmitter == null || topologySubmitter.equals("")) 
{
+LOG.error("Cannot determine user for topology {}.  Will 
skip scheduling this topology", td.getName());
+continue;
+}
--- End diff --

removed confusing comment


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063631#comment-15063631
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r48000168
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/SchedulingResult.java ---
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Map;
+
+/**
+ * This class serves as a mechanism to return results and messages from a 
scheduling strategy to the Resource Aware Scheduler
+ */
+public class SchedulingResult {
+
+//contains the result for the attempted scheduling
+private Map> 
schedulingResultMap = null;
+
+//status of scheduling the topology e.g. success or fail?
+private SchedulingStatus status = null;
+
+//arbitrary message to be returned when scheduling is done
+private String message = null;
+
+//error message returned is something went wrong
+private String errorMessage = null;
+
+private static final Logger LOG = 
LoggerFactory.getLogger(SchedulingResult.class);
+
+private SchedulingResult(SchedulingStatus status, Map> schedulingResultMap, String message, String 
errorMessage) {
+this.status = status;
+this.schedulingResultMap = schedulingResultMap;
+this.message = message;
+this.errorMessage = errorMessage;
+}
+
+public static SchedulingResult failure(SchedulingStatus status, String 
errorMessage) {
+return new SchedulingResult(status, null, null, errorMessage);
+}
+
+public static SchedulingResult success(Map> schedulingResultMap) {
+return SchedulingResult.successWithMsg(schedulingResultMap, null);
+}
+
+public static SchedulingResult successWithMsg(Map> schedulingResultMap, String message) {
+if (schedulingResultMap == null) {
+throw new IllegalStateException("Cannot declare scheduling 
success without providing a non null scheduling map!");
+}
+return new SchedulingResult(SchedulingStatus.SUCCESS, 
schedulingResultMap, message, null);
+}
+
+public SchedulingStatus getStatus() {
+return this.status;
+}
+
+public String getMessage() {
+return this.message;
+}
+
+public String getErrorMessage() {
+return this.errorMessage;
+}
--- End diff --

I am not sure if this is redundant but originally I thought errorMessage 
might want to set to contain more technical details of what problem actually 
occured and message will contain a simplified version of what happened to be 
displayed on the ui under scheduler status


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each us

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063630#comment-15063630
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r4792
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/eviction/DefaultEvictionStrategy.java
 ---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.eviction;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Map;
+
+public class DefaultEvictionStrategy implements IEvictionStrategy {
+private static final Logger LOG = LoggerFactory
+.getLogger(DefaultEvictionStrategy.class);
+
+private Topologies topologies;
+private Cluster cluster;
+private Map userMap;
+private RAS_Nodes nodes;
+
+@Override
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes) {
+this.topologies = topologies;
+this.cluster = cluster;
+this.userMap = userMap;
+this.nodes = nodes;
+}
+
+@Override
+public boolean makeSpaceForTopo(TopologyDetails td) {
+LOG.debug("attempting to make space for topo {} from user {}", 
td.getName(), td.getTopologySubmitter());
+User submitter = this.userMap.get(td.getTopologySubmitter());
+if (submitter.getCPUResourceGuaranteed() == null || 
submitter.getMemoryResourceGuaranteed() == null) {
+return false;
+}
+double cpuNeeded = td.getTotalRequestedCpu() / 
submitter.getCPUResourceGuaranteed();
+double memoryNeeded = (td.getTotalRequestedMemOffHeap() + 
td.getTotalRequestedMemOnHeap()) / submitter.getMemoryResourceGuaranteed();
+
+User evictUser = 
this.findUserWithHighestAverageResourceUtilAboveGuarantee();
+//user has enough resource under his or her resource guarantee to 
schedule topology
+if ((1.0 - submitter.getCPUResourcePoolUtilization()) >= cpuNeeded 
&& (1.0 - submitter.getMemoryResourcePoolUtilization()) >= memoryNeeded) {
+if (evictUser != null) {
+TopologyDetails topologyEvict = 
evictUser.getRunningTopologyWithLowestPriority();
+evictTopology(topologyEvict);
+return true;
+}
+} else {
+if (evictUser != null) {
+if ((evictUser.getResourcePoolAverageUtilization() - 1.0) 
> (cpuNeeded + (submitter.getResourcePoolAverageUtilization() - 1.0))) {
+TopologyDetails topologyEvict = 
evictUser.getRunningTopologyWithLowestPriority();
+evictTopology(topologyEvict);
+return true;
+}
+}
+}
+//See if there is a lower priority topology that can be evicted 
from the current user
+for (TopologyDetails topo : submitter.getTopologiesRunning()) {
+//check to if there is a topology with a lower priority we can 
evict
+if (topo.getTopologyPriority() > td.getTopologyPriority()) {
+evictTopology(topo);
+return true;
+}
+}
--- End diff --

well topologies might not actually be evicted if the checkpointed scheduler 
state is restored


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063622#comment-15063622
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47999760
  
--- Diff: storm-core/src/jvm/backtype/storm/scheduler/Cluster.java ---
@@ -430,12 +457,44 @@ public SupervisorDetails getSupervisorById(String 
nodeId) {
 }
 
 /**
+ * set assignments for cluster
+ */
+public void setAssignments(Map 
newAssignments) {
+this.assignments = new HashMap();
+for (Map.Entry entry : 
newAssignments.entrySet()) {
--- End diff --

sure changed to:
this.assignments = new HashMap(newAssignments.size());



> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063608#comment-15063608
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47999139
  
--- Diff: docs/documentation/Resource_Aware_Scheduler_overview.md ---
@@ -0,0 +1,227 @@
+# Introduction
+
+The purpose of this document is to provide a description of the Resource 
Aware Scheduler for the Storm distributed real-time computation system.  This 
document will provide you with both a high level description of the resource 
aware scheduler in Storm
+
+## Using Resource Aware Scheduler
+
+The user can switch to using the Resource Aware Scheduler by setting the 
following in *conf/storm.yaml*
+
+storm.scheduler: 
“backtype.storm.scheduler.resource.ResourceAwareScheduler”
+
+
+## API Overview
+
+For a Storm Topology, the user can now specify the amount of resources a 
topology component (i.e. Spout or Bolt) is required to run a single instance of 
the component.  The user can specify the resource requirement for a topology 
component by using the following API calls.
+
+### Setting Memory Requirement
+
+API to set component memory requirement:
+
+public T setMemoryLoad(Number onHeap, Number offHeap)
+
+Parameters:
+* Number onHeap – The amount of on heap memory an instance of this 
component will consume in megabytes
+* Number offHeap – The amount of off heap memory an instance of this 
component will consume in megabytes
+
+The user also has to option to just specify the on heap memory requirement 
if the component does not have an off heap memory need.
+
+public T setMemoryLoad(Number onHeap)
+
+Parameters:
+* Number onHeap – The amount of on heap memory an instance of this 
component will consume
+
+If no value is provided for offHeap, 0.0 will be used. If no value is 
provided for onHeap, or if the API is never called for a component, the default 
value will be used.
+
+Example of Usage:
+
+SpoutDeclarer s1 = builder.setSpout("word", new TestWordSpout(), 10);
+s1.setMemoryLoad(1024.0, 512.0);
+builder.setBolt("exclaim1", new ExclamationBolt(), 3)
+.shuffleGrouping("word").setMemoryLoad(512.0);
+
+The entire memory requested for this topology is 16.5 GB. That is from 10 
spouts with 1GB on heap memory and 0.5 GB off heap memory each and 3 bolts with 
0.5 GB on heap memory each.
+
+### Setting CPU Requirement
+
+
+API to set component CPU requirement:
+
+public T setCPULoad(Double amount)
+
+Parameters:
+* Number amount – The amount of on CPU an instance of this component will 
consume.
--- End diff --

well currently in the implementation we are assuming each executor only has 
one task.  We perhaps need to consider in the future how to handle executors 
will multiple tasks.  Since its not clear the resource usages when multiple 
tasks are executed serially in an executor


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> prio

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063603#comment-15063603
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47998998
  
--- Diff: docs/documentation/Resource_Aware_Scheduler_overview.md ---
@@ -0,0 +1,227 @@
+# Introduction
+
+The purpose of this document is to provide a description of the Resource 
Aware Scheduler for the Storm distributed real-time computation system.  This 
document will provide you with both a high level description of the resource 
aware scheduler in Storm
+
+## Using Resource Aware Scheduler
+
+The user can switch to using the Resource Aware Scheduler by setting the 
following in *conf/storm.yaml*
+
+storm.scheduler: 
“backtype.storm.scheduler.resource.ResourceAwareScheduler”
+
+
+## API Overview
+
+For a Storm Topology, the user can now specify the amount of resources a 
topology component (i.e. Spout or Bolt) is required to run a single instance of 
the component.  The user can specify the resource requirement for a topology 
component by using the following API calls.
+
+### Setting Memory Requirement
+
+API to set component memory requirement:
+
+public T setMemoryLoad(Number onHeap, Number offHeap)
+
+Parameters:
+* Number onHeap – The amount of on heap memory an instance of this 
component will consume in megabytes
+* Number offHeap – The amount of off heap memory an instance of this 
component will consume in megabytes
--- End diff --

Yup for each Storm task 


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063602#comment-15063602
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47998913
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestUtilsForResourceAwareScheduler.java
 ---
@@ -0,0 +1,290 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.generated.Bolt;
+import backtype.storm.generated.SpoutSpec;
+import backtype.storm.generated.StormTopology;
+import backtype.storm.generated.TopologySummary;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.OutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.BoltDeclarer;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.SpoutDeclarer;
+import backtype.storm.topology.TopologyBuilder;
+import backtype.storm.topology.base.BaseRichBolt;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+import backtype.storm.tuple.Tuple;
+import backtype.storm.tuple.Values;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+public class TestUtilsForResourceAwareScheduler {
+private static final Logger LOG = 
LoggerFactory.getLogger(TestUtilsForResourceAwareScheduler.class);
+
+public static List getListOfTopologies(Config config) 
{
+
+List topos = new LinkedList();
+
+topos.add(TestUtilsForResourceAwareScheduler.getTopology("topo-1", 
config, 5, 15, 1, 1, Time.currentTimeSecs() - 2, 20));
--- End diff --

will fix


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priori

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063598#comment-15063598
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47998805
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestUtilsForResourceAwareScheduler.java
 ---
@@ -0,0 +1,290 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.generated.Bolt;
+import backtype.storm.generated.SpoutSpec;
+import backtype.storm.generated.StormTopology;
+import backtype.storm.generated.TopologySummary;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.OutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.BoltDeclarer;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.SpoutDeclarer;
+import backtype.storm.topology.TopologyBuilder;
+import backtype.storm.topology.base.BaseRichBolt;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+import backtype.storm.tuple.Tuple;
+import backtype.storm.tuple.Values;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
--- End diff --

will remove


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063597#comment-15063597
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47998792
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestUtilsForResourceAwareScheduler.java
 ---
@@ -0,0 +1,290 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.generated.Bolt;
+import backtype.storm.generated.SpoutSpec;
+import backtype.storm.generated.StormTopology;
+import backtype.storm.generated.TopologySummary;
--- End diff --

will remove


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063596#comment-15063596
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47998771
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestUser.java ---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import org.junit.Assert;
+import org.junit.Test;
--- End diff --

This is used.  Need for all the @Test annotations to specify its a junit 
test


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063593#comment-15063593
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47998726
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063588#comment-15063588
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47998594
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063582#comment-15063582
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47998460
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063580#comment-15063580
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47998401
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063568#comment-15063568
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47997668
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063566#comment-15063566
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47997596
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063563#comment-15063563
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47997537
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063562#comment-15063562
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47997302
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063560#comment-15063560
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47997239
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063558#comment-15063558
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47997212
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063557#comment-15063557
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47997185
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063556#comment-15063556
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47997169
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063548#comment-15063548
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47997034
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063547#comment-15063547
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47997006
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails top

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063543#comment-15063543
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47996885
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
--- End diff --

will remove


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarant

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063539#comment-15063539
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47996829
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
--- End diff --

will remove


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063540#comment-15063540
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47996840
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
--- End diff --

will remove


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063535#comment-15063535
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47996761
  
--- Diff: storm-core/test/jvm/backtype/storm/TestConfigValidate.java ---
@@ -628,9 +628,89 @@ public void TestImpersonationAclUserEntryValidator() 
throws InvocationTargetExce
 } catch (IllegalArgumentException Ex) {
 }
 }
+}
+
+@Test
+public void TestResourceAwareSchedulerUserPool() {
+TestConfig config = new TestConfig();
+Collection passCases = new LinkedList();
+Collection failCases = new LinkedList();
+
+Map> passCase1 = new HashMap>();
+passCase1.put("jerry", new HashMap());
+passCase1.put("bobby", new HashMap());
+passCase1.put("derek", new HashMap());
+
+passCase1.get("jerry").put("cpu", 1);
+passCase1.get("jerry").put("memory", 20148);
+passCase1.get("bobby").put("cpu", 2);
+passCase1.get("bobby").put("memory", 40148);
+passCase1.get("derek").put("cpu", 3);
+passCase1.get("derek").put("memory", 60148);
+
+passCases.add(passCase1);
+
+for (Object value : passCases) {
+config.put(TestConfig.TEST_MAP_CONFIG_7, value);
+ConfigValidation.validateFields(config, TestConfig.class);
+}
+
+Map> failCase1 = new HashMap>();
+failCase1.put("jerry", new HashMap());
+failCase1.put("bobby", new HashMap());
+failCase1.put("derek", new HashMap());
+
+failCase1.get("jerry").put("cpu", 1);
+failCase1.get("jerry").put("memory", 20148);
+failCase1.get("bobby").put("cpu", 2);
+failCase1.get("bobby").put("memory", 40148);
+failCase1.get("derek").put("cpu", 3);
+
+Map> failCase2 = new HashMap>();
+failCase2.put("jerry", new HashMap());
+failCase2.put("bobby", new HashMap());
+failCase2.put("derek", new HashMap());
+failCase2.get("bobby").put("cpu", 2);
+failCase2.get("bobby").put("memory", 40148);
+failCase2.get("derek").put("cpu", 3);
+failCase2.get("derek").put("memory", 60148);
+
+failCases.add(failCase1);
+failCases.add(failCase2);
+
+for (Object value : failCases) {
+try {
+config.put(TestConfig.TEST_MAP_CONFIG_7, value);
+ConfigValidation.validateFields(config, TestConfig.class);
+Assert.fail("Expected Exception not Thrown for value: " + 
value);
+} catch (IllegalArgumentException Ex) {
+}
+}
+}
+
+@Test
+public void TestImplementsClassValidator() {
+TestConfig config = new TestConfig();
+Collection passCases = new LinkedList();
+Collection failCases = new LinkedList();
 
+
passCases.add("backtype.storm.networktopography.DefaultRackDNSToSwitchMapping");
 
+for (Object value : passCases) {
+config.put(TestConfig.TEST_MAP_CONFIG_8, value);
+ConfigValidation.validateFields(config, TestConfig.class);
+}
 
+failCases.add("backtype.storm.nimbus.NimbusInfo");
+failCases.add(null);
--- End diff --

will add


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063530#comment-15063530
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47996648
  
--- Diff: storm-core/test/jvm/backtype/storm/TestConfigValidate.java ---
@@ -628,9 +628,89 @@ public void TestImpersonationAclUserEntryValidator() 
throws InvocationTargetExce
 } catch (IllegalArgumentException Ex) {
 }
 }
+}
+
+@Test
+public void TestResourceAwareSchedulerUserPool() {
+TestConfig config = new TestConfig();
+Collection passCases = new LinkedList();
+Collection failCases = new LinkedList();
+
+Map> passCase1 = new HashMap>();
+passCase1.put("jerry", new HashMap());
+passCase1.put("bobby", new HashMap());
+passCase1.put("derek", new HashMap());
+
+passCase1.get("jerry").put("cpu", 1);
+passCase1.get("jerry").put("memory", 20148);
+passCase1.get("bobby").put("cpu", 2);
+passCase1.get("bobby").put("memory", 40148);
+passCase1.get("derek").put("cpu", 3);
+passCase1.get("derek").put("memory", 60148);
+
+passCases.add(passCase1);
+
+for (Object value : passCases) {
+config.put(TestConfig.TEST_MAP_CONFIG_7, value);
+ConfigValidation.validateFields(config, TestConfig.class);
+}
+
+Map> failCase1 = new HashMap>();
+failCase1.put("jerry", new HashMap());
+failCase1.put("bobby", new HashMap());
+failCase1.put("derek", new HashMap());
+
+failCase1.get("jerry").put("cpu", 1);
+failCase1.get("jerry").put("memory", 20148);
+failCase1.get("bobby").put("cpu", 2);
+failCase1.get("bobby").put("memory", 40148);
+failCase1.get("derek").put("cpu", 3);
--- End diff --

will add


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063527#comment-15063527
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47996525
  
--- Diff: storm-core/test/jvm/backtype/storm/TestConfigValidate.java ---
@@ -628,9 +628,89 @@ public void TestImpersonationAclUserEntryValidator() 
throws InvocationTargetExce
 } catch (IllegalArgumentException Ex) {
 }
 }
+}
+
+@Test
+public void TestResourceAwareSchedulerUserPool() {
+TestConfig config = new TestConfig();
+Collection passCases = new LinkedList();
+Collection failCases = new LinkedList();
+
+Map> passCase1 = new HashMap>();
+passCase1.put("jerry", new HashMap());
+passCase1.put("bobby", new HashMap());
+passCase1.put("derek", new HashMap());
+
+passCase1.get("jerry").put("cpu", 1);
+passCase1.get("jerry").put("memory", 20148);
+passCase1.get("bobby").put("cpu", 2);
+passCase1.get("bobby").put("memory", 40148);
+passCase1.get("derek").put("cpu", 3);
+passCase1.get("derek").put("memory", 60148);
+
+passCases.add(passCase1);
+
+for (Object value : passCases) {
+config.put(TestConfig.TEST_MAP_CONFIG_7, value);
+ConfigValidation.validateFields(config, TestConfig.class);
+}
--- End diff --

ok will get rid of loop


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063525#comment-15063525
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on the pull request:

https://github.com/apache/storm/pull/921#issuecomment-165679412
  
OK, review pass done.

I am wondering about slots.  We have cpu and memory resources, but we also 
have slots that are a kind of virtual/logical resource.  How will RAS behave if 
the number of slots runs low?  Do the eviction/priority strategies work the 
same way as if there were plenty of slots?

This might have been covered before, but does this change how many slots a 
cluster admin should configure per supervisor?


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063523#comment-15063523
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47996336
  
--- Diff: 
storm-core/src/jvm/backtype/storm/validation/ConfigValidation.java ---
@@ -489,17 +489,63 @@ public  void validateField(String name, Object o) {
 public static class PacemakerAuthTypeValidator extends Validator {
 @Override
 public void validateField(String name, Object o) {
-if(o == null) {
-throw new IllegalArgumentException( "Field " + name + " 
must be set.");
+if (o == null) {
+throw new IllegalArgumentException("Field " + name + " 
must be set.");
+}
+
+if (o instanceof String &&
+(((String) o).equals("NONE") ||
+((String) o).equals("DIGEST") ||
+((String) o).equals("KERBEROS"))) {
+return;
+}
+throw new IllegalArgumentException("Field " + name + " must be 
one of \"NONE\", \"DIGEST\", or \"KERBEROS\"");
+}
+}
--- End diff --

Yup, but I am not sure why git thinks the diff is this.  The spacing seems 
to be identical.  Git seems to have gotten confused which part was changed


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063515#comment-15063515
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47996150
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/scheduling/IStrategy.java
 ---
@@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.scheduling;
+
+import java.util.Collection;
+import java.util.Map;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.scheduler.resource.RAS_Node;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.SchedulingResult;
+import backtype.storm.scheduler.resource.User;
--- End diff --

will remove


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063514#comment-15063514
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47996124
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/priority/ISchedulingPriorityStrategy.java
 ---
@@ -16,22 +16,26 @@
  * limitations under the License.
  */
 
-package backtype.storm.scheduler.resource.strategies;
-
-import java.util.Collection;
-import java.util.Map;
+package backtype.storm.scheduler.resource.strategies.priority;
 
+import backtype.storm.scheduler.Cluster;
 import backtype.storm.scheduler.Topologies;
-import backtype.storm.scheduler.ExecutorDetails;
 import backtype.storm.scheduler.TopologyDetails;
-import backtype.storm.scheduler.WorkerSlot;
-import backtype.storm.scheduler.resource.RAS_Node;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
 
-/**
- * An interface to for implementing different scheduling strategies for 
the resource aware scheduling
- * In the future stategies will be pluggable
- */
-public interface IStrategy {
+import java.util.Map;
+
+public interface ISchedulingPriorityStrategy {
+
+/**
+ * initializes
+ */
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes);
--- End diff --

well we cannot get all the topologydetails from cluster.  And we would have 
to recreate all the node objects from cluster which I think might be 
unnecessary since we already do it in the scheduler so why do this computation 
again uneccessarily


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063508#comment-15063508
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47996047
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/priority/DefaultSchedulingPriorityStrategy.java
 ---
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.priority;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+
+public class DefaultSchedulingPriorityStrategy implements 
ISchedulingPriorityStrategy {
+private static final Logger LOG = LoggerFactory
+.getLogger(DefaultSchedulingPriorityStrategy.class);
+
+private Topologies topologies;
+private Cluster cluster;
+private Map userMap;
+private RAS_Nodes nodes;
--- End diff --

will remove


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063505#comment-15063505
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47996031
  
--- Diff: docs/documentation/Resource_Aware_Scheduler_overview.md ---
@@ -0,0 +1,227 @@
+# Introduction
+
+The purpose of this document is to provide a description of the Resource 
Aware Scheduler for the Storm distributed real-time computation system.  This 
document will provide you with both a high level description of the resource 
aware scheduler in Storm
+
+## Using Resource Aware Scheduler
+
+The user can switch to using the Resource Aware Scheduler by setting the 
following in *conf/storm.yaml*
+
+storm.scheduler: 
“backtype.storm.scheduler.resource.ResourceAwareScheduler”
+
+
+## API Overview
+
+For a Storm Topology, the user can now specify the amount of resources a 
topology component (i.e. Spout or Bolt) is required to run a single instance of 
the component.  The user can specify the resource requirement for a topology 
component by using the following API calls.
+
+### Setting Memory Requirement
+
+API to set component memory requirement:
+
+public T setMemoryLoad(Number onHeap, Number offHeap)
+
+Parameters:
+* Number onHeap – The amount of on heap memory an instance of this 
component will consume in megabytes
+* Number offHeap – The amount of off heap memory an instance of this 
component will consume in megabytes
+
+The user also has to option to just specify the on heap memory requirement 
if the component does not have an off heap memory need.
+
+public T setMemoryLoad(Number onHeap)
+
+Parameters:
+* Number onHeap – The amount of on heap memory an instance of this 
component will consume
+
+If no value is provided for offHeap, 0.0 will be used. If no value is 
provided for onHeap, or if the API is never called for a component, the default 
value will be used.
+
+Example of Usage:
+
+SpoutDeclarer s1 = builder.setSpout("word", new TestWordSpout(), 10);
+s1.setMemoryLoad(1024.0, 512.0);
+builder.setBolt("exclaim1", new ExclamationBolt(), 3)
+.shuffleGrouping("word").setMemoryLoad(512.0);
+
+The entire memory requested for this topology is 16.5 GB. That is from 10 
spouts with 1GB on heap memory and 0.5 GB off heap memory each and 3 bolts with 
0.5 GB on heap memory each.
+
+### Setting CPU Requirement
+
+
+API to set component CPU requirement:
+
+public T setCPULoad(Double amount)
+
+Parameters:
+* Number amount – The amount of on CPU an instance of this component will 
consume.
--- End diff --

It seems "instance" here means Storm Executor/parallelism/threads.  This is 
a different "instance" than a Storm Task.  Are instances different between 
memory resources and cpu resources? If so, do we need to clarify?


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is eno

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063507#comment-15063507
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47996045
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/priority/DefaultSchedulingPriorityStrategy.java
 ---
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.priority;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+
+public class DefaultSchedulingPriorityStrategy implements 
ISchedulingPriorityStrategy {
+private static final Logger LOG = LoggerFactory
+.getLogger(DefaultSchedulingPriorityStrategy.class);
+
+private Topologies topologies;
--- End diff --

will remove


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063502#comment-15063502
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47995984
  
--- Diff: docs/documentation/Resource_Aware_Scheduler_overview.md ---
@@ -0,0 +1,227 @@
+# Introduction
+
+The purpose of this document is to provide a description of the Resource 
Aware Scheduler for the Storm distributed real-time computation system.  This 
document will provide you with both a high level description of the resource 
aware scheduler in Storm
+
+## Using Resource Aware Scheduler
+
+The user can switch to using the Resource Aware Scheduler by setting the 
following in *conf/storm.yaml*
+
+storm.scheduler: 
“backtype.storm.scheduler.resource.ResourceAwareScheduler”
+
+
+## API Overview
+
+For a Storm Topology, the user can now specify the amount of resources a 
topology component (i.e. Spout or Bolt) is required to run a single instance of 
the component.  The user can specify the resource requirement for a topology 
component by using the following API calls.
+
+### Setting Memory Requirement
+
+API to set component memory requirement:
+
+public T setMemoryLoad(Number onHeap, Number offHeap)
+
+Parameters:
+* Number onHeap – The amount of on heap memory an instance of this 
component will consume in megabytes
+* Number offHeap – The amount of off heap memory an instance of this 
component will consume in megabytes
--- End diff --

What is meant by instance for memory resources? Will each Storm Task be an 
instance?


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063504#comment-15063504
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47996002
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/eviction/DefaultEvictionStrategy.java
 ---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.eviction;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Map;
+
+public class DefaultEvictionStrategy implements IEvictionStrategy {
+private static final Logger LOG = LoggerFactory
+.getLogger(DefaultEvictionStrategy.class);
+
+private Topologies topologies;
+private Cluster cluster;
+private Map userMap;
+private RAS_Nodes nodes;
+
+@Override
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes) {
+this.topologies = topologies;
+this.cluster = cluster;
+this.userMap = userMap;
+this.nodes = nodes;
+}
+
+@Override
+public boolean makeSpaceForTopo(TopologyDetails td) {
+LOG.debug("attempting to make space for topo {} from user {}", 
td.getName(), td.getTopologySubmitter());
+User submitter = this.userMap.get(td.getTopologySubmitter());
+if (submitter.getCPUResourceGuaranteed() == null || 
submitter.getMemoryResourceGuaranteed() == null) {
+return false;
+}
+double cpuNeeded = td.getTotalRequestedCpu() / 
submitter.getCPUResourceGuaranteed();
+double memoryNeeded = (td.getTotalRequestedMemOffHeap() + 
td.getTotalRequestedMemOnHeap()) / submitter.getMemoryResourceGuaranteed();
+
+User evictUser = 
this.findUserWithHighestAverageResourceUtilAboveGuarantee();
+//user has enough resource under his or her resource guarantee to 
schedule topology
+if ((1.0 - submitter.getCPUResourcePoolUtilization()) >= cpuNeeded 
&& (1.0 - submitter.getMemoryResourcePoolUtilization()) >= memoryNeeded) {
+if (evictUser != null) {
+TopologyDetails topologyEvict = 
evictUser.getRunningTopologyWithLowestPriority();
+evictTopology(topologyEvict);
+return true;
+}
+} else {
+if (evictUser != null) {
+if ((evictUser.getResourcePoolAverageUtilization() - 1.0) 
> (cpuNeeded + (submitter.getResourcePoolAverageUtilization() - 1.0))) {
+TopologyDetails topologyEvict = 
evictUser.getRunningTopologyWithLowestPriority();
+evictTopology(topologyEvict);
+return true;
+}
+}
+}
+//See if there is a lower priority topology that can be evicted 
from the current user
+for (TopologyDetails topo : submitter.getTopologiesRunning()) {
+//check to if there is a topology with a lower priority we can 
evict
+if (topo.getTopologyPriority() > td.getTopologyPriority()) {
+evictTopology(topo);
+return true;
+}
+}
+return false;
+}
+
+private void evictTopology(TopologyDetails topologyEvict) {
+Collection workersToEvict = 
this.cluster.getUsedSlotsByTopologyId(topologyEvict.ge

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063495#comment-15063495
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47995699
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestUtilsForResourceAwareScheduler.java
 ---
@@ -0,0 +1,290 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.generated.Bolt;
+import backtype.storm.generated.SpoutSpec;
+import backtype.storm.generated.StormTopology;
+import backtype.storm.generated.TopologySummary;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.OutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.BoltDeclarer;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.SpoutDeclarer;
+import backtype.storm.topology.TopologyBuilder;
+import backtype.storm.topology.base.BaseRichBolt;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+import backtype.storm.tuple.Tuple;
+import backtype.storm.tuple.Values;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+public class TestUtilsForResourceAwareScheduler {
+private static final Logger LOG = 
LoggerFactory.getLogger(TestUtilsForResourceAwareScheduler.class);
+
+public static List getListOfTopologies(Config config) 
{
+
+List topos = new LinkedList();
+
+topos.add(TestUtilsForResourceAwareScheduler.getTopology("topo-1", 
config, 5, 15, 1, 1, Time.currentTimeSecs() - 2, 20));
--- End diff --

Same comment about `currentTimeSecs`


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  S

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063492#comment-15063492
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47995673
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestUtilsForResourceAwareScheduler.java
 ---
@@ -0,0 +1,290 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.generated.Bolt;
+import backtype.storm.generated.SpoutSpec;
+import backtype.storm.generated.StormTopology;
+import backtype.storm.generated.TopologySummary;
--- End diff --

Unused


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063493#comment-15063493
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47995677
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestUtilsForResourceAwareScheduler.java
 ---
@@ -0,0 +1,290 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.generated.Bolt;
+import backtype.storm.generated.SpoutSpec;
+import backtype.storm.generated.StormTopology;
+import backtype.storm.generated.TopologySummary;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.OutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.BoltDeclarer;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.SpoutDeclarer;
+import backtype.storm.topology.TopologyBuilder;
+import backtype.storm.topology.base.BaseRichBolt;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+import backtype.storm.tuple.Tuple;
+import backtype.storm.tuple.Values;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
--- End diff --

both unused


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063490#comment-15063490
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47995488
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/eviction/DefaultEvictionStrategy.java
 ---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.eviction;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Map;
+
+public class DefaultEvictionStrategy implements IEvictionStrategy {
+private static final Logger LOG = LoggerFactory
+.getLogger(DefaultEvictionStrategy.class);
+
+private Topologies topologies;
+private Cluster cluster;
+private Map userMap;
+private RAS_Nodes nodes;
+
+@Override
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes) {
+this.topologies = topologies;
+this.cluster = cluster;
+this.userMap = userMap;
+this.nodes = nodes;
+}
+
+@Override
+public boolean makeSpaceForTopo(TopologyDetails td) {
+LOG.debug("attempting to make space for topo {} from user {}", 
td.getName(), td.getTopologySubmitter());
+User submitter = this.userMap.get(td.getTopologySubmitter());
+if (submitter.getCPUResourceGuaranteed() == null || 
submitter.getMemoryResourceGuaranteed() == null) {
+return false;
+}
+double cpuNeeded = td.getTotalRequestedCpu() / 
submitter.getCPUResourceGuaranteed();
+double memoryNeeded = (td.getTotalRequestedMemOffHeap() + 
td.getTotalRequestedMemOnHeap()) / submitter.getMemoryResourceGuaranteed();
+
+User evictUser = 
this.findUserWithHighestAverageResourceUtilAboveGuarantee();
+//user has enough resource under his or her resource guarantee to 
schedule topology
--- End diff --

how about I change it to "check if user has enough resource under his or 
her resource guarantee to schedule topology" since the next statement is a null 
check and putting the comment above that could be confusing


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resourc

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063485#comment-15063485
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47995363
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestUser.java ---
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import org.junit.Assert;
+import org.junit.Test;
--- End diff --

unused


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063483#comment-15063483
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47995347
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails topo1 = 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063473#comment-15063473
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47995263
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/eviction/DefaultEvictionStrategy.java
 ---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.eviction;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Map;
+
+public class DefaultEvictionStrategy implements IEvictionStrategy {
+private static final Logger LOG = LoggerFactory
+.getLogger(DefaultEvictionStrategy.class);
+
+private Topologies topologies;
+private Cluster cluster;
+private Map userMap;
+private RAS_Nodes nodes;
+
+@Override
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes) {
+this.topologies = topologies;
+this.cluster = cluster;
+this.userMap = userMap;
+this.nodes = nodes;
+}
+
+@Override
+public boolean makeSpaceForTopo(TopologyDetails td) {
+LOG.debug("attempting to make space for topo {} from user {}", 
td.getName(), td.getTopologySubmitter());
+User submitter = this.userMap.get(td.getTopologySubmitter());
+if (submitter.getCPUResourceGuaranteed() == null || 
submitter.getMemoryResourceGuaranteed() == null) {
+return false;
+}
+double cpuNeeded = td.getTotalRequestedCpu() / 
submitter.getCPUResourceGuaranteed();
+double memoryNeeded = (td.getTotalRequestedMemOffHeap() + 
td.getTotalRequestedMemOnHeap()) / submitter.getMemoryResourceGuaranteed();
--- End diff --

Well if getCPUResourceGuaranteed() or getTotalRequestedMemOnHeap() is zero, 
cpuNeeded and memoryNeeded will be infinity.  The logic in the rest of the code 
should be able to handle this situation correctly, but perhaps as a 
optimization we should just return false early since that is going to be the 
outcome anyways but returning false early will not have to go through the 
additional logic


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063470#comment-15063470
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47995098
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails topo1 = 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063469#comment-15063469
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47995063
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails topo1 = 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063465#comment-15063465
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47994923
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails topo1 = 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063457#comment-15063457
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47994712
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails topo1 = 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063425#comment-15063425
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47993642
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/eviction/DefaultEvictionStrategy.java
 ---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.eviction;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Map;
+
+public class DefaultEvictionStrategy implements IEvictionStrategy {
+private static final Logger LOG = LoggerFactory
+.getLogger(DefaultEvictionStrategy.class);
+
+private Topologies topologies;
+private Cluster cluster;
+private Map userMap;
+private RAS_Nodes nodes;
+
+@Override
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes) {
+this.topologies = topologies;
+this.cluster = cluster;
+this.userMap = userMap;
+this.nodes = nodes;
+}
+
+@Override
+public boolean makeSpaceForTopo(TopologyDetails td) {
+LOG.debug("attempting to make space for topo {} from user {}", 
td.getName(), td.getTopologySubmitter());
+User submitter = this.userMap.get(td.getTopologySubmitter());
+if (submitter.getCPUResourceGuaranteed() == null || 
submitter.getMemoryResourceGuaranteed() == null) {
+return false;
--- End diff --

well its normal operations, since a user can have no resource guarantee and 
still submit a topology


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063423#comment-15063423
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47993566
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/eviction/IEvictionStrategy.java
 ---
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.eviction;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+
+import java.util.Map;
+
+public interface IEvictionStrategy {
+
+/**
+ * Initialization
+ */
+public void prepare(Topologies topologies, Cluster cluster, 
Map userMap, RAS_Nodes nodes);
--- End diff --

I think so, since we could write a IEviction Strategy that needs to take 
into account what topologies have been submitted.  We want to give this 
interface all the possible information the scheduler can.  Also we don't want 
to come back and have to add the topologies parameter latter if we need it for 
another strategy we implemented.


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063418#comment-15063418
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47993443
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/eviction/DefaultEvictionStrategy.java
 ---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource.strategies.eviction;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.scheduler.resource.RAS_Nodes;
+import backtype.storm.scheduler.resource.User;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Map;
+
+public class DefaultEvictionStrategy implements IEvictionStrategy {
+private static final Logger LOG = LoggerFactory
+.getLogger(DefaultEvictionStrategy.class);
+
+private Topologies topologies;
--- End diff --

Well its not used in the DefaultEvictionStrategy, but its passed in part of 
the prepare method that implements the IEviction interface, but I guess we can 
remove it


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063419#comment-15063419
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47993464
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails topo1 = 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063413#comment-15063413
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47993313
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails topo1 = 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063409#comment-15063409
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47993245
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails topo1 = 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063402#comment-15063402
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47992997
  
--- Diff: storm-core/src/jvm/backtype/storm/scheduler/resource/User.java ---
@@ -0,0 +1,346 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.TopologyDetails;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeSet;
+
+public class User {
+private String userId;
+//Topologies yet to be scheduled sorted by priority for each user
+private TreeSet pendingQueue = new 
TreeSet(new PQsortByPriorityAndSubmittionTime());
+
+//Topologies yet to be scheduled sorted by priority for each user
+private TreeSet runningQueue = new 
TreeSet(new PQsortByPriorityAndSubmittionTime());
+
+//Topologies that was attempted to be scheduled but wasn't successull
+private TreeSet attemptedQueue = new 
TreeSet(new PQsortByPriorityAndSubmittionTime());
+
+private TreeSet invalidQueue = new 
TreeSet(new PQsortByPriorityAndSubmittionTime());
+
+private Map resourcePool = new HashMap();
+
+private static final Logger LOG = LoggerFactory.getLogger(User.class);
+
+public User(String userId) {
+this.userId = userId;
+}
+
+public User(String userId, Map resourcePool) {
+this(userId);
+if (resourcePool != null) {
+this.resourcePool.putAll(resourcePool);
+}
+if (this.resourcePool.get("cpu") == null) {
+this.resourcePool.put("cpu", 0.0);
+}
+if (this.resourcePool.get("memory") == null) {
+this.resourcePool.put("memory", 0.0);
+}
+}
+
+public User getCopy() {
+User newUser = new User(this.userId, this.resourcePool);
+for (TopologyDetails topo : this.pendingQueue) {
+newUser.addTopologyToPendingQueue(topo);
+}
+for (TopologyDetails topo : this.runningQueue) {
+newUser.addTopologyToRunningQueue(topo);
+}
+for (TopologyDetails topo : this.attemptedQueue) {
+newUser.addTopologyToAttemptedQueue(topo);
+}
+for (TopologyDetails topo : this.invalidQueue) {
+newUser.addTopologyToInvalidQueue(topo);
+}
+return newUser;
+}
+
+public String getId() {
+return this.userId;
+}
+
+public void addTopologyToPendingQueue(TopologyDetails topo, Cluster 
cluster) {
+this.pendingQueue.add(topo);
+if (cluster != null) {
+cluster.setStatus(topo.getId(), "Scheduling Pending");
+}
+}
+
+public void addTopologyToPendingQueue(TopologyDetails topo) {
+this.addTopologyToPendingQueue(topo, null);
+}
+
+public void addTopologyToRunningQueue(TopologyDetails topo, Cluster 
cluster) {
+this.runningQueue.add(topo);
+if (cluster != null) {
+cluster.setStatus(topo.getId(), "Fully Scheduled");
+}
+}
+
+public void addTopologyToRunningQueue(TopologyDetails topo) {
+this.addTopologyToRunningQueue(topo, null);
+}
+
+public Set getTopologiesPending() {
+TreeSet ret = new TreeSet(new 
PQsortByPriorityAndSubmittionTime());
+  

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063399#comment-15063399
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47992974
  
--- Diff: storm-core/src/jvm/backtype/storm/scheduler/resource/User.java ---
@@ -0,0 +1,346 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.TopologyDetails;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
--- End diff --

will remove


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063397#comment-15063397
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47992968
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails topo1 = 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063393#comment-15063393
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47992825
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/ResourceUtils.java ---
@@ -130,4 +137,57 @@ private static void debugMessage(String memoryType, 
String Com, Map topologyConf
 Com, 
topologyConf.get(Config.TOPOLOGY_COMPONENT_CPU_PCORE_PERCENT));
 }
 }
+
+/**
+ * print scheduling for debug purposes
+ * @param cluster
+ * @param topologies
+ */
+public static String printScheduling(Cluster cluster, Topologies 
topologies) {
+StringBuilder str = new StringBuilder();
+Map>>> schedulingMap = new HashMap>>>();
+for (TopologyDetails topo : topologies.getTopologies()) {
+if (cluster.getAssignmentById(topo.getId()) != null) {
+for (Map.Entry entry : 
cluster.getAssignmentById(topo.getId()).getExecutorToSlot().entrySet()) {
+WorkerSlot slot = entry.getValue();
+String nodeId = slot.getNodeId();
+ExecutorDetails exec = entry.getKey();
+if (schedulingMap.containsKey(nodeId) == false) {
+schedulingMap.put(nodeId, new HashMap>>());
+}
+if 
(schedulingMap.get(nodeId).containsKey(topo.getId()) == false) {
+schedulingMap.get(nodeId).put(topo.getId(), new 
HashMap>());
+}
+if 
(schedulingMap.get(nodeId).get(topo.getId()).containsKey(slot) == false) {
+
schedulingMap.get(nodeId).get(topo.getId()).put(slot, new 
LinkedList());
+}
+
schedulingMap.get(nodeId).get(topo.getId()).get(slot).add(exec);
+}
+}
+}
+
+for (Map.Entry>>> entry : schedulingMap.entrySet()) {
+if (cluster.getSupervisorById(entry.getKey()) != null) {
+str.append("/** Node: " + 
cluster.getSupervisorById(entry.getKey()).getHost() + "-" + entry.getKey() + " 
**/\n");
+} else {
+str.append("/** Node: Unknown may be dead -" + 
entry.getKey() + " **/\n");
+}
+for (Map.Entry>> topo_sched : 
schedulingMap.get(entry.getKey()).entrySet()) {
+str.append("\t-->Topology: " + topo_sched.getKey() + "\n");
+for (Map.Entry> ws 
: topo_sched.getValue().entrySet()) {
+str.append("\t\t->Slot [" + ws.getKey().getPort() + "] 
-> " + ws.getValue() + "\n");
+}
+}
+}
+return str.toString();
+}
+
+public static String printScheduling(RAS_Nodes nodes) {
+String ret="";
+for (RAS_Node node : nodes.getNodes()) {
+ret += "Node: " + node.getHostname() + "\n";
+ret += "-> " + node.getTopoIdTousedSlots() + "\n";
+}
+return ret;
+}
--- End diff --

ya sure will remove


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don'

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063390#comment-15063390
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47992746
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/ResourceUtils.java ---
@@ -130,4 +137,57 @@ private static void debugMessage(String memoryType, 
String Com, Map topologyConf
 Com, 
topologyConf.get(Config.TOPOLOGY_COMPONENT_CPU_PCORE_PERCENT));
 }
 }
+
+/**
+ * print scheduling for debug purposes
+ * @param cluster
+ * @param topologies
+ */
+public static String printScheduling(Cluster cluster, Topologies 
topologies) {
+StringBuilder str = new StringBuilder();
+Map>>> schedulingMap = new HashMap>>>();
+for (TopologyDetails topo : topologies.getTopologies()) {
+if (cluster.getAssignmentById(topo.getId()) != null) {
+for (Map.Entry entry : 
cluster.getAssignmentById(topo.getId()).getExecutorToSlot().entrySet()) {
+WorkerSlot slot = entry.getValue();
+String nodeId = slot.getNodeId();
+ExecutorDetails exec = entry.getKey();
+if (schedulingMap.containsKey(nodeId) == false) {
--- End diff --

will fix


> Add priorities and per user resource guarantees to Resource Aware Scheduler
> ---
>
> Key: STORM-898
> URL: https://issues.apache.org/jira/browse/STORM-898
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Boyang Jerry Peng
> Attachments: Resource Aware Scheduler for Storm.pdf
>
>
> In a multi-tenant environment we would like to be able to give individual 
> users a guarantee of how much CPU/Memory/Network they will be able to use in 
> a cluster.  We would also like to know which topologies a user feels are the 
> most important to keep running if there are not enough resources to run all 
> of their topologies.
> Each user should be able to specify if their topology is production, staging, 
> or development. Within each of those categories a user should be able to give 
> a topology a priority, 0 to 10 with 10 being the highest priority (or 
> something like this).
> If there are not enough resources on a cluster to run a topology assume this 
> topology is running using resources and find the user that is most over their 
> guaranteed resources.  Shoot the lowest priority topology for that user, and 
> repeat until, this topology is able to run, or this topology would be the one 
> shot.   Ideally we don't actually shoot anything until we know that we would 
> have made enough room.
> If the cluster is over-subscribed and everyone is under their guarantee, and 
> this topology would not put the user over their guarantee.  Shoot the lowest 
> priority topology in this workers resource pool until there is enough room to 
> run the topology or this topology is the one that would be shot.  We might 
> also want to think about what to do if we are going to shoot a production 
> topology in an oversubscribed case, and perhaps we can shoot a non-production 
> topology instead even if the other user is not over their guarantee.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063389#comment-15063389
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47992737
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/ResourceAwareScheduler.java
 ---
@@ -128,56 +308,121 @@ private void updateSupervisorsResources(Cluster 
cluster, Topologies topologies)
 cluster.setSupervisorsResourcesMap(supervisors_resources);
 }
 
-private Map getUserConf() {
-Map ret = new HashMap();
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB));
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB));
-return ret;
+public User getUser(String user) {
+return this.userMap.get(user);
+}
+
+public Map getUserMap() {
+return this.userMap;
 }
 
 /**
- * print scheduling for debug purposes
- * @param cluster
+ * Intialize scheduling and running queues
+ *
  * @param topologies
+ * @param cluster
  */
-public String printScheduling(Cluster cluster, Topologies topologies) {
-StringBuilder str = new StringBuilder();
-Map>>> schedulingMap = new HashMap>>>();
-for (TopologyDetails topo : topologies.getTopologies()) {
-if (cluster.getAssignmentById(topo.getId()) != null) {
-for (Map.Entry entry : 
cluster.getAssignmentById(topo.getId()).getExecutorToSlot().entrySet()) {
-WorkerSlot slot = entry.getValue();
-String nodeId = slot.getNodeId();
-ExecutorDetails exec = entry.getKey();
-if (schedulingMap.containsKey(nodeId) == false) {
-schedulingMap.put(nodeId, new HashMap>>());
-}
-if 
(schedulingMap.get(nodeId).containsKey(topo.getId()) == false) {
-schedulingMap.get(nodeId).put(topo.getId(), new 
HashMap>());
-}
-if 
(schedulingMap.get(nodeId).get(topo.getId()).containsKey(slot) == false) {
-
schedulingMap.get(nodeId).get(topo.getId()).put(slot, new 
LinkedList());
-}
-
schedulingMap.get(nodeId).get(topo.getId()).get(slot).add(exec);
+private void initUsers(Topologies topologies, Cluster cluster) {
+this.userMap = new HashMap();
+Map> userResourcePools = 
this.getUserResourcePools();
+LOG.debug("userResourcePools: {}", userResourcePools);
+
+for (TopologyDetails td : topologies.getTopologies()) {
+//Get user that submitted topology.  If topology submitter is 
null or empty string, the topologySubmitter
+//will be set to anonymous
+String topologySubmitter = td.getTopologySubmitter();
+//additional safety check to make sure that topologySubmitter 
is going to be a valid value
+if (topologySubmitter == null || topologySubmitter.equals("")) 
{
+LOG.error("Cannot determine user for topology {}.  Will 
skip scheduling this topology", td.getName());
+continue;
+}
+if (!this.userMap.containsKey(topologySubmitter)) {
+this.userMap.put(topologySubmitter, new 
User(topologySubmitter, userResourcePools.get(topologySubmitter)));
+}
+if (cluster.getUnassignedExecutors(td).size() > 0) {
+LOG.debug("adding td: {} to pending queue", td.getName());
+
this.userMap.get(topologySubmitter).addTopologyToPendingQueue(td);
+} else {
+LOG.debug("adding td: {} to running queue with existing 
status: {}", td.getName(), cluster.getStatusMap().get(td.getId()));
+
this.userMap.get(topologySubmitter).addTopologyToRunningQueue(td);
+if (cluster.getStatusMap().get(td.getId()) == null || 
cluster.getStatusMap().get(td.getId()).equals("")) {
+cluster.setStatus(td.getId(), "Fully Scheduled");
 }
 }
 }
+}
 
-for (Map.Entry>>> entry : schedulingMap.entrySet()) {
-if (cluster.getSupervisorById(entry.getKey()) != null) {
-str.append("/** 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063387#comment-15063387
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47992667
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/ResourceAwareScheduler.java
 ---
@@ -128,56 +308,121 @@ private void updateSupervisorsResources(Cluster 
cluster, Topologies topologies)
 cluster.setSupervisorsResourcesMap(supervisors_resources);
 }
 
-private Map getUserConf() {
-Map ret = new HashMap();
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB));
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB));
-return ret;
+public User getUser(String user) {
+return this.userMap.get(user);
+}
+
+public Map getUserMap() {
+return this.userMap;
 }
 
 /**
- * print scheduling for debug purposes
- * @param cluster
+ * Intialize scheduling and running queues
+ *
  * @param topologies
+ * @param cluster
  */
-public String printScheduling(Cluster cluster, Topologies topologies) {
-StringBuilder str = new StringBuilder();
-Map>>> schedulingMap = new HashMap>>>();
-for (TopologyDetails topo : topologies.getTopologies()) {
-if (cluster.getAssignmentById(topo.getId()) != null) {
-for (Map.Entry entry : 
cluster.getAssignmentById(topo.getId()).getExecutorToSlot().entrySet()) {
-WorkerSlot slot = entry.getValue();
-String nodeId = slot.getNodeId();
-ExecutorDetails exec = entry.getKey();
-if (schedulingMap.containsKey(nodeId) == false) {
-schedulingMap.put(nodeId, new HashMap>>());
-}
-if 
(schedulingMap.get(nodeId).containsKey(topo.getId()) == false) {
-schedulingMap.get(nodeId).put(topo.getId(), new 
HashMap>());
-}
-if 
(schedulingMap.get(nodeId).get(topo.getId()).containsKey(slot) == false) {
-
schedulingMap.get(nodeId).get(topo.getId()).put(slot, new 
LinkedList());
-}
-
schedulingMap.get(nodeId).get(topo.getId()).get(slot).add(exec);
+private void initUsers(Topologies topologies, Cluster cluster) {
+this.userMap = new HashMap();
+Map> userResourcePools = 
this.getUserResourcePools();
+LOG.debug("userResourcePools: {}", userResourcePools);
+
+for (TopologyDetails td : topologies.getTopologies()) {
+//Get user that submitted topology.  If topology submitter is 
null or empty string, the topologySubmitter
+//will be set to anonymous
+String topologySubmitter = td.getTopologySubmitter();
+//additional safety check to make sure that topologySubmitter 
is going to be a valid value
+if (topologySubmitter == null || topologySubmitter.equals("")) 
{
+LOG.error("Cannot determine user for topology {}.  Will 
skip scheduling this topology", td.getName());
+continue;
+}
+if (!this.userMap.containsKey(topologySubmitter)) {
+this.userMap.put(topologySubmitter, new 
User(topologySubmitter, userResourcePools.get(topologySubmitter)));
+}
+if (cluster.getUnassignedExecutors(td).size() > 0) {
+LOG.debug("adding td: {} to pending queue", td.getName());
+
this.userMap.get(topologySubmitter).addTopologyToPendingQueue(td);
+} else {
+LOG.debug("adding td: {} to running queue with existing 
status: {}", td.getName(), cluster.getStatusMap().get(td.getId()));
+
this.userMap.get(topologySubmitter).addTopologyToRunningQueue(td);
+if (cluster.getStatusMap().get(td.getId()) == null || 
cluster.getStatusMap().get(td.getId()).equals("")) {
+cluster.setStatus(td.getId(), "Fully Scheduled");
 }
 }
 }
+}
 
-for (Map.Entry>>> entry : schedulingMap.entrySet()) {
-if (cluster.getSupervisorById(entry.getKey()) != null) {
-str.append("/** 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063385#comment-15063385
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47992609
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails topo1 = 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063384#comment-15063384
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user d2r commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47992606
  
--- Diff: 
storm-core/test/jvm/backtype/storm/scheduler/resource/TestResourceAwareScheduler.java
 ---
@@ -0,0 +1,1166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package backtype.storm.scheduler.resource;
+
+import backtype.storm.Config;
+import backtype.storm.scheduler.Cluster;
+import backtype.storm.scheduler.ExecutorDetails;
+import backtype.storm.scheduler.INimbus;
+import backtype.storm.scheduler.IScheduler;
+import backtype.storm.scheduler.SchedulerAssignment;
+import backtype.storm.scheduler.SchedulerAssignmentImpl;
+import backtype.storm.scheduler.SupervisorDetails;
+import backtype.storm.scheduler.Topologies;
+import backtype.storm.scheduler.TopologyDetails;
+import backtype.storm.scheduler.WorkerSlot;
+import backtype.storm.utils.Time;
+import backtype.storm.utils.Utils;
+import backtype.storm.validation.ConfigValidation;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+public class TestResourceAwareScheduler {
+
+private static final int NUM_SUPS = 20;
+private static final int NUM_WORKERS_PER_SUP = 4;
+private final String TOPOLOGY_SUBMITTER = "jerry";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(TestResourceAwareScheduler.class);
+
+@Test
+public void TestReadInResourceAwareSchedulerUserPools() {
+
+Map fromFile = 
Utils.findAndReadConfigFile("user-resource-pools.yaml", false);
+LOG.info("fromFile: {}", fromFile);
+ConfigValidation.validateFields(fromFile);
+}
+
+@Test
+public void TestTopologySortedInCorrectOrder() {
+INimbus iNimbus = new 
TestUtilsForResourceAwareScheduler.INimbusTest();
+Map resourceMap = new HashMap();
+resourceMap.put(Config.SUPERVISOR_CPU_CAPACITY, 100.0);
+resourceMap.put(Config.SUPERVISOR_MEMORY_CAPACITY_MB, 1024.0);
+Map supMap = 
TestUtilsForResourceAwareScheduler.genSupervisors(20, 4, resourceMap);
+Config config = new Config();
+config.putAll(Utils.readDefaultConfig());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_EVICTION_STRATEGY, 
backtype.storm.scheduler.resource.strategies.eviction.DefaultEvictionStrategy.class.getName());
+config.put(Config.RESOURCE_AWARE_SCHEDULER_PRIORITY_STRATEGY, 
backtype.storm.scheduler.resource.strategies.priority.DefaultSchedulingPriorityStrategy.class.getName());
+config.put(Config.TOPOLOGY_SCHEDULER_STRATEGY, 
backtype.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class.getName());
+
+config.put(Config.TOPOLOGY_SUBMITTER_USER, TOPOLOGY_SUBMITTER);
+
+Map> resourceUserPool = new 
HashMap>();
+resourceUserPool.put("jerry", new HashMap());
+resourceUserPool.get("jerry").put("cpu", 1000);
+resourceUserPool.get("jerry").put("memory", 8192.0);
+
+resourceUserPool.put("bobby", new HashMap());
+resourceUserPool.get("bobby").put("cpu", 1.0);
+resourceUserPool.get("bobby").put("memory", 32768);
+
+resourceUserPool.put("derek", new HashMap());
+resourceUserPool.get("derek").put("cpu", 5000.0);
+resourceUserPool.get("derek").put("memory", 16384.0);
+
+config.put(Config.RESOURCE_AWARE_SCHEDULER_USER_POOLS, 
resourceUserPool);
+
+TopologyDetails topo1 = 

[jira] [Commented] (STORM-898) Add priorities and per user resource guarantees to Resource Aware Scheduler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063382#comment-15063382
 ] 

ASF GitHub Bot commented on STORM-898:
--

Github user jerrypeng commented on a diff in the pull request:

https://github.com/apache/storm/pull/921#discussion_r47992580
  
--- Diff: 
storm-core/src/jvm/backtype/storm/scheduler/resource/ResourceAwareScheduler.java
 ---
@@ -128,56 +308,121 @@ private void updateSupervisorsResources(Cluster 
cluster, Topologies topologies)
 cluster.setSupervisorsResourcesMap(supervisors_resources);
 }
 
-private Map getUserConf() {
-Map ret = new HashMap();
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB));
-ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB,
-(Double) 
_conf.get(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB));
-return ret;
+public User getUser(String user) {
+return this.userMap.get(user);
+}
+
+public Map getUserMap() {
+return this.userMap;
 }
 
 /**
- * print scheduling for debug purposes
- * @param cluster
+ * Intialize scheduling and running queues
+ *
  * @param topologies
+ * @param cluster
  */
-public String printScheduling(Cluster cluster, Topologies topologies) {
-StringBuilder str = new StringBuilder();
-Map>>> schedulingMap = new HashMap>>>();
-for (TopologyDetails topo : topologies.getTopologies()) {
-if (cluster.getAssignmentById(topo.getId()) != null) {
-for (Map.Entry entry : 
cluster.getAssignmentById(topo.getId()).getExecutorToSlot().entrySet()) {
-WorkerSlot slot = entry.getValue();
-String nodeId = slot.getNodeId();
-ExecutorDetails exec = entry.getKey();
-if (schedulingMap.containsKey(nodeId) == false) {
-schedulingMap.put(nodeId, new HashMap>>());
-}
-if 
(schedulingMap.get(nodeId).containsKey(topo.getId()) == false) {
-schedulingMap.get(nodeId).put(topo.getId(), new 
HashMap>());
-}
-if 
(schedulingMap.get(nodeId).get(topo.getId()).containsKey(slot) == false) {
-
schedulingMap.get(nodeId).get(topo.getId()).put(slot, new 
LinkedList());
-}
-
schedulingMap.get(nodeId).get(topo.getId()).get(slot).add(exec);
+private void initUsers(Topologies topologies, Cluster cluster) {
+this.userMap = new HashMap();
+Map> userResourcePools = 
this.getUserResourcePools();
+LOG.debug("userResourcePools: {}", userResourcePools);
+
+for (TopologyDetails td : topologies.getTopologies()) {
+//Get user that submitted topology.  If topology submitter is 
null or empty string, the topologySubmitter
+//will be set to anonymous
+String topologySubmitter = td.getTopologySubmitter();
+//additional safety check to make sure that topologySubmitter 
is going to be a valid value
+if (topologySubmitter == null || topologySubmitter.equals("")) 
{
+LOG.error("Cannot determine user for topology {}.  Will 
skip scheduling this topology", td.getName());
+continue;
+}
+if (!this.userMap.containsKey(topologySubmitter)) {
+this.userMap.put(topologySubmitter, new 
User(topologySubmitter, userResourcePools.get(topologySubmitter)));
+}
+if (cluster.getUnassignedExecutors(td).size() > 0) {
+LOG.debug("adding td: {} to pending queue", td.getName());
+
this.userMap.get(topologySubmitter).addTopologyToPendingQueue(td);
+} else {
+LOG.debug("adding td: {} to running queue with existing 
status: {}", td.getName(), cluster.getStatusMap().get(td.getId()));
+
this.userMap.get(topologySubmitter).addTopologyToRunningQueue(td);
+if (cluster.getStatusMap().get(td.getId()) == null || 
cluster.getStatusMap().get(td.getId()).equals("")) {
+cluster.setStatus(td.getId(), "Fully Scheduled");
 }
 }
 }
+}
 
-for (Map.Entry>>> entry : schedulingMap.entrySet()) {
-if (cluster.getSupervisorById(entry.getKey()) != null) {
-str.append("/**