Re: Review Request 30710: add mesos role feature

[lozh...@ebay.com zhang]

> On Feb. 17, 2015, 11:09 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/configuration/Resources.java, 
> > line 174
> > 
> >
> > Prefer using guava `Ordering` [1] instead. It will avoid excessive 
> > collection copies.
> > 
> > [1] - https://code.google.com/p/guava-libraries/wiki/OrderingExplained

Will change


> On Feb. 17, 2015, 11:09 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/configuration/Resources.java, 
> > line 139
> > 
> >
> > PORTS should be treated the same as other Mesos resources as far as 
> > framework role goes. We need to set the framework role on a matching port 
> > `Resource`.

Will Change


- lozh...@ebay.com


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review72793
---


On Feb. 13, 2015, 10 a.m., lozh...@ebay.com zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30710/
> ---
> 
> (Updated Feb. 13, 2015, 10 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> ## Problems
> 
> We are from eBay platform team. Previously, we used marathon to generate 
> Jenkins master instance in dedicated vms and recieve resource offer from same 
> dedicated vms. For the details, please refer to
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
> 
> Now, we found Aurora is more stable and powerful. We are moving from Marathon 
> to Aurora. During the move, we found there is no mesos role in Aurora now. 
> But we need use mesos role way to solve the problem in section "Frameworks 
> stopped receiving offers after a while" of the given url.
> 
> Here is a snippet of the problem description:
> 
> *We noticed occurred after we used Marathon to create the initial set of CI 
> masters. As those CI masters started registering themselves as frameworks, 
> Marathon stopped receiving any offers from Mesos; essentially, no new CI 
> masters could be launched. Let’s start with Marathon. In the DRF model, it 
> was unfair to treat Marathon in the same bucket/role alongside hundreds of 
> connected Jenkins frameworks. After launching all these Jenkins frameworks, 
> Marathon had a large resource share and Mesos would aggressively offer 
> resources to frameworks that were using little or no resources. Marathon was 
> placed last in priority and got starved out.*
> 
> *We decided to define a dedicated Mesos role for Marathon and to have all of 
> the Mesos slaves that were reserved for Jenkins master instances support that 
> Mesos role. Jenkins frameworks were left with the default role “*”.*This 
> solved the problem – Mesos offered resources per role and hence Marathon 
> never got starved out. A framework with a special role will get resource 
> offers from both slaves supporting that special role and also from the 
> default role “*”.* However, since we were using placement constraints, 
> Marathon accepted resource offers only from slaves that supported both the 
> role and the placement constraints.*
> ## Solution
> 
> So we add role feature is the source code to solve the problem in same way: 
> When accept a resource offer, Aurora will send back the needed resources to 
> Mesos with the mesos role in resource offer.
> 
> How to configure the Mesos role:
> 1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
> 
> We change the test cases according code change. Each changed test case is 
> green
> Merge https://github.com/zhanglong2015/incubator-aurora
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
> 1a158b4e0be94762ad0480e8ce74b19bacc90c97 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> 31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58 
>   src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 
> b5a3140e3560f790d1db496dca3c2ee0dc96a195 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  d0994203b5650f44ca2eb32e1e2aa61875163854 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> 5340d651b298ec8aa079e73d6d2f652fdf876293 
>   src/main/java/org/apache/aurora/scheduler/mesos/OfferWrapper.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/ResourceWrapper.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/sche

Re: Review Request 30710: add mesos role feature

[lozh...@ebay.com zhang]

> On Feb. 17, 2015, 11:09 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/configuration/Resources.java, 
> > line 114
> > 
> >
> > This is breaking Resources/Offer encapsulation and is not addressing 
> > the long term possiblility of making role-biased scheduling decisions. 
> > 
> > A possible solution could be "de-anonymize" `Resources` by adding a 
> > `String role` to it. The `role` would be left 
> > `Optional.absent()` any time an "anonymous" `Resources` instance is 
> > required or populated to the respective framework role when a role-specific 
> > resources are initilized. This would obviously require refactoring 
> > `Resources` to support internal mapping of resource vectors by role.
> > 
> > The above would also eliminate the need for "wrapper" classes and will 
> > keep the door open for future role-based scheduling optimizations.
> > 
> > I am curious what others think here?
> 
> lozh...@ebay.com zhang wrote:
> May I have your skype No.? let's discuss it

The wrapper classes is used to track which resources have bean allocated. 
Do you mean tracking the resouce allocation in Resources instead? if so, there 
are some issues:
1.Noramally each task will has two Resources instance: one for task itself, one 
for executor. So is unable to track the resource allcation in Resources 
instance, unless we use thread local variable or something like this.

I think another solution is:
1.keep Resources/Offer unchanged, so the Resources/Offer encapsulation is not 
broken.
2.add a new class ResourceAllocationTracker which is used to track the resource 
allocation, this class's constructor should accept a Offer as argument.
3.pass the instance of ResourceAllocationTracker to Resources.toResourceList as 
new parameter
4.in Resources.toResourceList, evertytime a resource is allocated, will be 
saved in ResourceAllocationTracker instance.


- lozh...@ebay.com


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review72793
---


On Feb. 13, 2015, 10 a.m., lozh...@ebay.com zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30710/
> ---
> 
> (Updated Feb. 13, 2015, 10 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> ## Problems
> 
> We are from eBay platform team. Previously, we used marathon to generate 
> Jenkins master instance in dedicated vms and recieve resource offer from same 
> dedicated vms. For the details, please refer to
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
> 
> Now, we found Aurora is more stable and powerful. We are moving from Marathon 
> to Aurora. During the move, we found there is no mesos role in Aurora now. 
> But we need use mesos role way to solve the problem in section "Frameworks 
> stopped receiving offers after a while" of the given url.
> 
> Here is a snippet of the problem description:
> 
> *We noticed occurred after we used Marathon to create the initial set of CI 
> masters. As those CI masters started registering themselves as frameworks, 
> Marathon stopped receiving any offers from Mesos; essentially, no new CI 
> masters could be launched. Let’s start with Marathon. In the DRF model, it 
> was unfair to treat Marathon in the same bucket/role alongside hundreds of 
> connected Jenkins frameworks. After launching all these Jenkins frameworks, 
> Marathon had a large resource share and Mesos would aggressively offer 
> resources to frameworks that were using little or no resources. Marathon was 
> placed last in priority and got starved out.*
> 
> *We decided to define a dedicated Mesos role for Marathon and to have all of 
> the Mesos slaves that were reserved for Jenkins master instances support that 
> Mesos role. Jenkins frameworks were left with the default role “*”.*This 
> solved the problem – Mesos offered resources per role and hence Marathon 
> never got starved out. A framework with a special role will get resource 
> offers from both slaves supporting that special role and also from the 
> default role “*”.* However, since we were using placement constraints, 
> Marathon accepted resource offers only from slaves that supported both the 
> role and the placement constraints.*
> ## Solution
> 
> So we add role feature is the source code to solve the problem in same way: 
> When accept a resource offer, Aurora will send back the needed resources to 
> Me

Re: Review Request 30710: add mesos role feature

[lozh...@ebay.com zhang]

> On Feb. 17, 2015, 11:09 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/configuration/Resources.java, 
> > line 114
> > 
> >
> > This is breaking Resources/Offer encapsulation and is not addressing 
> > the long term possiblility of making role-biased scheduling decisions. 
> > 
> > A possible solution could be "de-anonymize" `Resources` by adding a 
> > `String role` to it. The `role` would be left 
> > `Optional.absent()` any time an "anonymous" `Resources` instance is 
> > required or populated to the respective framework role when a role-specific 
> > resources are initilized. This would obviously require refactoring 
> > `Resources` to support internal mapping of resource vectors by role.
> > 
> > The above would also eliminate the need for "wrapper" classes and will 
> > keep the door open for future role-based scheduling optimizations.
> > 
> > I am curious what others think here?

May I have your skype No.? let's discuss it


- lozh...@ebay.com


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review72793
---


On Feb. 13, 2015, 10 a.m., lozh...@ebay.com zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30710/
> ---
> 
> (Updated Feb. 13, 2015, 10 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> ## Problems
> 
> We are from eBay platform team. Previously, we used marathon to generate 
> Jenkins master instance in dedicated vms and recieve resource offer from same 
> dedicated vms. For the details, please refer to
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
> 
> Now, we found Aurora is more stable and powerful. We are moving from Marathon 
> to Aurora. During the move, we found there is no mesos role in Aurora now. 
> But we need use mesos role way to solve the problem in section "Frameworks 
> stopped receiving offers after a while" of the given url.
> 
> Here is a snippet of the problem description:
> 
> *We noticed occurred after we used Marathon to create the initial set of CI 
> masters. As those CI masters started registering themselves as frameworks, 
> Marathon stopped receiving any offers from Mesos; essentially, no new CI 
> masters could be launched. Let’s start with Marathon. In the DRF model, it 
> was unfair to treat Marathon in the same bucket/role alongside hundreds of 
> connected Jenkins frameworks. After launching all these Jenkins frameworks, 
> Marathon had a large resource share and Mesos would aggressively offer 
> resources to frameworks that were using little or no resources. Marathon was 
> placed last in priority and got starved out.*
> 
> *We decided to define a dedicated Mesos role for Marathon and to have all of 
> the Mesos slaves that were reserved for Jenkins master instances support that 
> Mesos role. Jenkins frameworks were left with the default role “*”.*This 
> solved the problem – Mesos offered resources per role and hence Marathon 
> never got starved out. A framework with a special role will get resource 
> offers from both slaves supporting that special role and also from the 
> default role “*”.* However, since we were using placement constraints, 
> Marathon accepted resource offers only from slaves that supported both the 
> role and the placement constraints.*
> ## Solution
> 
> So we add role feature is the source code to solve the problem in same way: 
> When accept a resource offer, Aurora will send back the needed resources to 
> Mesos with the mesos role in resource offer.
> 
> How to configure the Mesos role:
> 1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
> 
> We change the test cases according code change. Each changed test case is 
> green
> Merge https://github.com/zhanglong2015/incubator-aurora
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
> 1a158b4e0be94762ad0480e8ce74b19bacc90c97 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> 31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58 
>   src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 
> b5a3140e3560f790d1db496dca3c2ee0dc96a195 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  d0994203b5650f44ca2eb32e1e2aa61875163854 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> 5340d651b298ec8aa079e73d6d2

Re: Review Request 30710: add mesos role feature

[Aurora ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review72862
---


This patch does not apply cleanly on master (4b43305), do you need to rebase?

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 13, 2015, 10 a.m., lozh...@ebay.com zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30710/
> ---
> 
> (Updated Feb. 13, 2015, 10 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> ## Problems
> 
> We are from eBay platform team. Previously, we used marathon to generate 
> Jenkins master instance in dedicated vms and recieve resource offer from same 
> dedicated vms. For the details, please refer to
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
> 
> Now, we found Aurora is more stable and powerful. We are moving from Marathon 
> to Aurora. During the move, we found there is no mesos role in Aurora now. 
> But we need use mesos role way to solve the problem in section "Frameworks 
> stopped receiving offers after a while" of the given url.
> 
> Here is a snippet of the problem description:
> 
> *We noticed occurred after we used Marathon to create the initial set of CI 
> masters. As those CI masters started registering themselves as frameworks, 
> Marathon stopped receiving any offers from Mesos; essentially, no new CI 
> masters could be launched. Let’s start with Marathon. In the DRF model, it 
> was unfair to treat Marathon in the same bucket/role alongside hundreds of 
> connected Jenkins frameworks. After launching all these Jenkins frameworks, 
> Marathon had a large resource share and Mesos would aggressively offer 
> resources to frameworks that were using little or no resources. Marathon was 
> placed last in priority and got starved out.*
> 
> *We decided to define a dedicated Mesos role for Marathon and to have all of 
> the Mesos slaves that were reserved for Jenkins master instances support that 
> Mesos role. Jenkins frameworks were left with the default role “*”.*This 
> solved the problem – Mesos offered resources per role and hence Marathon 
> never got starved out. A framework with a special role will get resource 
> offers from both slaves supporting that special role and also from the 
> default role “*”.* However, since we were using placement constraints, 
> Marathon accepted resource offers only from slaves that supported both the 
> role and the placement constraints.*
> ## Solution
> 
> So we add role feature is the source code to solve the problem in same way: 
> When accept a resource offer, Aurora will send back the needed resources to 
> Mesos with the mesos role in resource offer.
> 
> How to configure the Mesos role:
> 1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
> 
> We change the test cases according code change. Each changed test case is 
> green
> Merge https://github.com/zhanglong2015/incubator-aurora
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
> 1a158b4e0be94762ad0480e8ce74b19bacc90c97 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> 31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58 
>   src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 
> b5a3140e3560f790d1db496dca3c2ee0dc96a195 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  d0994203b5650f44ca2eb32e1e2aa61875163854 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> 5340d651b298ec8aa079e73d6d2f652fdf876293 
>   src/main/java/org/apache/aurora/scheduler/mesos/OfferWrapper.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/ResourceWrapper.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
> e1c29747c9854cf75bf63f6f085cf40ca68989af 
>   src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java 
> 422d5a9a42310979752eb7282658316c2b772419 
>   src/test/java/org/apache/aurora/scheduler/configuration/ResourcesTest.java 
> d6febb8998e05257cabe8d193cefa0b6c79f197e 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
> 5f08d00d39f016af9bc296e517ad49b66ab5a8de 
>   src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
> 411a55a8d85f60bb2703468f2d69b64b2736eee4 
> 
> Diff: https://reviews.apache.org/r/30710/diff/
> 
> 
> Testing
> ---
> 
> :buildSrc:compileJava UP-

Re: Review Request 30710: add mesos role feature

[Bill Farner]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review72842
---


@ReviewBot retry

- Bill Farner


On Feb. 13, 2015, 10 a.m., lozh...@ebay.com zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30710/
> ---
> 
> (Updated Feb. 13, 2015, 10 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> ## Problems
> 
> We are from eBay platform team. Previously, we used marathon to generate 
> Jenkins master instance in dedicated vms and recieve resource offer from same 
> dedicated vms. For the details, please refer to
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
> 
> Now, we found Aurora is more stable and powerful. We are moving from Marathon 
> to Aurora. During the move, we found there is no mesos role in Aurora now. 
> But we need use mesos role way to solve the problem in section "Frameworks 
> stopped receiving offers after a while" of the given url.
> 
> Here is a snippet of the problem description:
> 
> *We noticed occurred after we used Marathon to create the initial set of CI 
> masters. As those CI masters started registering themselves as frameworks, 
> Marathon stopped receiving any offers from Mesos; essentially, no new CI 
> masters could be launched. Let’s start with Marathon. In the DRF model, it 
> was unfair to treat Marathon in the same bucket/role alongside hundreds of 
> connected Jenkins frameworks. After launching all these Jenkins frameworks, 
> Marathon had a large resource share and Mesos would aggressively offer 
> resources to frameworks that were using little or no resources. Marathon was 
> placed last in priority and got starved out.*
> 
> *We decided to define a dedicated Mesos role for Marathon and to have all of 
> the Mesos slaves that were reserved for Jenkins master instances support that 
> Mesos role. Jenkins frameworks were left with the default role “*”.*This 
> solved the problem – Mesos offered resources per role and hence Marathon 
> never got starved out. A framework with a special role will get resource 
> offers from both slaves supporting that special role and also from the 
> default role “*”.* However, since we were using placement constraints, 
> Marathon accepted resource offers only from slaves that supported both the 
> role and the placement constraints.*
> ## Solution
> 
> So we add role feature is the source code to solve the problem in same way: 
> When accept a resource offer, Aurora will send back the needed resources to 
> Mesos with the mesos role in resource offer.
> 
> How to configure the Mesos role:
> 1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
> 
> We change the test cases according code change. Each changed test case is 
> green
> Merge https://github.com/zhanglong2015/incubator-aurora
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
> 1a158b4e0be94762ad0480e8ce74b19bacc90c97 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> 31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58 
>   src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 
> b5a3140e3560f790d1db496dca3c2ee0dc96a195 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  d0994203b5650f44ca2eb32e1e2aa61875163854 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> 5340d651b298ec8aa079e73d6d2f652fdf876293 
>   src/main/java/org/apache/aurora/scheduler/mesos/OfferWrapper.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/ResourceWrapper.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
> e1c29747c9854cf75bf63f6f085cf40ca68989af 
>   src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java 
> 422d5a9a42310979752eb7282658316c2b772419 
>   src/test/java/org/apache/aurora/scheduler/configuration/ResourcesTest.java 
> d6febb8998e05257cabe8d193cefa0b6c79f197e 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
> 5f08d00d39f016af9bc296e517ad49b66ab5a8de 
>   src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
> 411a55a8d85f60bb2703468f2d69b64b2736eee4 
> 
> Diff: https://reviews.apache.org/r/30710/diff/
> 
> 
> Testing
> ---
> 
> :buildSrc:compileJava UP-TO-DATE
> :buildSrc:compileGroovy UP-TO-DATE
> :buildSrc:processResources UP-TO-DATE
> :buildSrc:classes UP-TO-DATE
> :buildSrc:jar UP-TO-DATE
> :buildS

Re: Review Request 30710: add mesos role feature

[Maxim Khutornenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review72793
---



src/main/java/org/apache/aurora/scheduler/configuration/Resources.java


This is breaking Resources/Offer encapsulation and is not addressing the 
long term possiblility of making role-biased scheduling decisions. 

A possible solution could be "de-anonymize" `Resources` by adding a 
`String role` to it. The `role` would be left 
`Optional.absent()` any time an "anonymous" `Resources` instance is 
required or populated to the respective framework role when a role-specific 
resources are initilized. This would obviously require refactoring `Resources` 
to support internal mapping of resource vectors by role.

The above would also eliminate the need for "wrapper" classes and will keep 
the door open for future role-based scheduling optimizations.

I am curious what others think here?



src/main/java/org/apache/aurora/scheduler/configuration/Resources.java


PORTS should be treated the same as other Mesos resources as far as 
framework role goes. We need to set the framework role on a matching port 
`Resource`.



src/main/java/org/apache/aurora/scheduler/configuration/Resources.java


Prefer using guava `Ordering` [1] instead. It will avoid excessive 
collection copies.

[1] - https://code.google.com/p/guava-libraries/wiki/OrderingExplained


- Maxim Khutornenko


On Feb. 13, 2015, 10 a.m., lozh...@ebay.com zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30710/
> ---
> 
> (Updated Feb. 13, 2015, 10 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> ## Problems
> 
> We are from eBay platform team. Previously, we used marathon to generate 
> Jenkins master instance in dedicated vms and recieve resource offer from same 
> dedicated vms. For the details, please refer to
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
> 
> Now, we found Aurora is more stable and powerful. We are moving from Marathon 
> to Aurora. During the move, we found there is no mesos role in Aurora now. 
> But we need use mesos role way to solve the problem in section "Frameworks 
> stopped receiving offers after a while" of the given url.
> 
> Here is a snippet of the problem description:
> 
> *We noticed occurred after we used Marathon to create the initial set of CI 
> masters. As those CI masters started registering themselves as frameworks, 
> Marathon stopped receiving any offers from Mesos; essentially, no new CI 
> masters could be launched. Let’s start with Marathon. In the DRF model, it 
> was unfair to treat Marathon in the same bucket/role alongside hundreds of 
> connected Jenkins frameworks. After launching all these Jenkins frameworks, 
> Marathon had a large resource share and Mesos would aggressively offer 
> resources to frameworks that were using little or no resources. Marathon was 
> placed last in priority and got starved out.*
> 
> *We decided to define a dedicated Mesos role for Marathon and to have all of 
> the Mesos slaves that were reserved for Jenkins master instances support that 
> Mesos role. Jenkins frameworks were left with the default role “*”.*This 
> solved the problem – Mesos offered resources per role and hence Marathon 
> never got starved out. A framework with a special role will get resource 
> offers from both slaves supporting that special role and also from the 
> default role “*”.* However, since we were using placement constraints, 
> Marathon accepted resource offers only from slaves that supported both the 
> role and the placement constraints.*
> ## Solution
> 
> So we add role feature is the source code to solve the problem in same way: 
> When accept a resource offer, Aurora will send back the needed resources to 
> Mesos with the mesos role in resource offer.
> 
> How to configure the Mesos role:
> 1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
> 
> We change the test cases according code change. Each changed test case is 
> green
> Merge https://github.com/zhanglong2015/incubator-aurora
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
> 1a158b4e0be94762ad0480e8ce74b19bacc90c97 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> 31aa2bbaab3d97875493ad75

Re: Review Request 30710: add mesos role feature

[Aurora ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review72113
---

Ship it!


Master (61e6c35) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 12, 2015, 6:12 a.m., lozh...@ebay.com zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30710/
> ---
> 
> (Updated Feb. 12, 2015, 6:12 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Bill Farner.
> 
> 
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> ## Problems
> 
> We are from eBay platform team. Previously, we used marathon to generate 
> Jenkins master instance in dedicated vms and recieve resource offer from same 
> dedicated vms. For the details, please refer to
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
> 
> Now, we found Aurora is more stable and powerful. We are moving from Marathon 
> to Aurora. During the move, we found there is no mesos role in Aurora now. 
> But we need use mesos role way to solve the problem in section "Frameworks 
> stopped receiving offers after a while" of the given url.
> 
> Here is a snippet of the problem description:
> 
> *We noticed occurred after we used Marathon to create the initial set of CI 
> masters. As those CI masters started registering themselves as frameworks, 
> Marathon stopped receiving any offers from Mesos; essentially, no new CI 
> masters could be launched. Let’s start with Marathon. In the DRF model, it 
> was unfair to treat Marathon in the same bucket/role alongside hundreds of 
> connected Jenkins frameworks. After launching all these Jenkins frameworks, 
> Marathon had a large resource share and Mesos would aggressively offer 
> resources to frameworks that were using little or no resources. Marathon was 
> placed last in priority and got starved out.*
> 
> *We decided to define a dedicated Mesos role for Marathon and to have all of 
> the Mesos slaves that were reserved for Jenkins master instances support that 
> Mesos role. Jenkins frameworks were left with the default role “*”.*This 
> solved the problem – Mesos offered resources per role and hence Marathon 
> never got starved out. A framework with a special role will get resource 
> offers from both slaves supporting that special role and also from the 
> default role “*”.* However, since we were using placement constraints, 
> Marathon accepted resource offers only from slaves that supported both the 
> role and the placement constraints.*
> ## Solution
> 
> So we add role feature is the source code to solve the problem in same way: 
> When accept a resource offer, Aurora will send back the needed resources to 
> Mesos with the mesos role in resource offer.
> 
> How to configure the Mesos role:
> 1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
> 
> We change the test cases according code change. Each changed test case is 
> green
> Merge https://github.com/zhanglong2015/incubator-aurora
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
> 1a158b4e0be94762ad0480e8ce74b19bacc90c97 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> 31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58 
>   src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 
> b5a3140e3560f790d1db496dca3c2ee0dc96a195 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  d0994203b5650f44ca2eb32e1e2aa61875163854 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> 5340d651b298ec8aa079e73d6d2f652fdf876293 
>   src/main/java/org/apache/aurora/scheduler/mesos/OfferWrapper.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/ResourceWrapper.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
> e1c29747c9854cf75bf63f6f085cf40ca68989af 
>   src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java 
> 422d5a9a42310979752eb7282658316c2b772419 
>   src/test/java/org/apache/aurora/scheduler/configuration/ResourcesTest.java 
> d6febb8998e05257cabe8d193cefa0b6c79f197e 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
> 5f08d00d39f016af9bc296e517ad49b66ab5a8de 
>   src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
> 411a55a8d85f60bb2703468f2d69b64b2736eee4 
> 
> Diff: https://reviews.apache.org/r/30710/diff/
> 
> 
> Testing
> ---
> 
> :buildSrc:compileJava UP-TO-DAT

Re: Review Request 30710: add mesos role feature

[lozh...@ebay.com zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/
---

(Updated Feb. 12, 2015, 6:12 a.m.)


Review request for Aurora, Joshua Cohen and Bill Farner.


Changes
---

1.Left the role unset in register framework message if no role specified in 
command line
2.For below case, will use the resource with role aurora first. If the resource 
with role aurora is not enough for task, then use resource with role "*" 
together. For example, a task need 5 CPU, then willl use "cpus(aurora):4 and 
cpus(*):1".
  "I am afraid the current approach is not going to work in a cluster with 
mesos slaves configured to offer multi-role resources. Consider an example 
where a slave is configured to offer both role-specific and general pool CPU:

--resources=cpus(aurora):4;cpus(*):2
"


Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109


Repository: aurora


Description
---

## Problems

We are from eBay platform team. Previously, we used marathon to generate 
Jenkins master instance in dedicated vms and recieve resource offer from same 
dedicated vms. For the details, please refer to
http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU

Now, we found Aurora is more stable and powerful. We are moving from Marathon 
to Aurora. During the move, we found there is no mesos role in Aurora now. But 
we need use mesos role way to solve the problem in section "Frameworks stopped 
receiving offers after a while" of the given url.

Here is a snippet of the problem description:

*We noticed occurred after we used Marathon to create the initial set of CI 
masters. As those CI masters started registering themselves as frameworks, 
Marathon stopped receiving any offers from Mesos; essentially, no new CI 
masters could be launched. Let’s start with Marathon. In the DRF model, it was 
unfair to treat Marathon in the same bucket/role alongside hundreds of 
connected Jenkins frameworks. After launching all these Jenkins frameworks, 
Marathon had a large resource share and Mesos would aggressively offer 
resources to frameworks that were using little or no resources. Marathon was 
placed last in priority and got starved out.*

*We decided to define a dedicated Mesos role for Marathon and to have all of 
the Mesos slaves that were reserved for Jenkins master instances support that 
Mesos role. Jenkins frameworks were left with the default role “*”.*This solved 
the problem – Mesos offered resources per role and hence Marathon never got 
starved out. A framework with a special role will get resource offers from both 
slaves supporting that special role and also from the default role “*”.* 
However, since we were using placement constraints, Marathon accepted resource 
offers only from slaves that supported both the role and the placement 
constraints.*
## Solution

So we add role feature is the source code to solve the problem in same way: 
When accept a resource offer, Aurora will send back the needed resources to 
Mesos with the mesos role in resource offer.

How to configure the Mesos role:
1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.

We change the test cases according code change. Each changed test case is green
Merge https://github.com/zhanglong2015/incubator-aurora


Diffs (updated)
-

  src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
1a158b4e0be94762ad0480e8ce74b19bacc90c97 
  src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58 
  src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 
b5a3140e3560f790d1db496dca3c2ee0dc96a195 
  
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
 d0994203b5650f44ca2eb32e1e2aa61875163854 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
5340d651b298ec8aa079e73d6d2f652fdf876293 
  src/main/java/org/apache/aurora/scheduler/mesos/OfferWrapper.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/ResourceWrapper.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
e1c29747c9854cf75bf63f6f085cf40ca68989af 
  src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java 
422d5a9a42310979752eb7282658316c2b772419 
  src/test/java/org/apache/aurora/scheduler/configuration/ResourcesTest.java 
d6febb8998e05257cabe8d193cefa0b6c79f197e 
  src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
5f08d00d39f016af9bc296e517ad49b66ab5a8de 
  src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
411a55a8d85f60bb2703468f2d69b64b2736eee4 

Diff: https://reviews.apache.org/r/30710/diff/


Testing
---

:buildSrc:compileJava UP-TO-DATE
:buildSrc:compileGroovy UP-TO-DATE
:buildSrc:processResources UP

Re: Review Request 30710: add mesos role feature

[Maxim Khutornenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review71754
---



src/main/java/org/apache/aurora/scheduler/configuration/Resources.java


I am afraid the current approach is not going to work in a cluster with 
mesos slaves configured to offer multi-role resources. Consider an example 
where a slave is configured to offer both role-specific and general pool CPU:
```
--resources=cpus(aurora):4;cpus(*):2
```

A correspondent offer resources section could be:
```
...
  "resources" :
  [
{
  "name": "cpus",
  "type": SCALAR,
  "scalar"  : { "value" : 4.0 },
  "role": "aurora",
},
{
  "name": "cpus",
  "type": SCALAR,
  "scalar"  : { "value" : 2.0 },
  "role": "*",
}
  ]
...
```
Given the above, a matching task with 6.0 CPU would require a TaskInfo with 
the same resources section (4.0-aurora, 2.0-*) or mesos would reject the task 
launch.

The current state of Aurora where a framework role is not configurable 
gives us the "luxury" of not supporting multi-role allocations. However, if we 
are going to support role-based allocations I think we should do it the proper 
way and fully support single and multi-role assignments. This is going to be 
especially critical when the mesos dynamic reservation [1] lands. 

[1] - 
https://docs.google.com/document/d/1e3j69pfBgtc8xM00DhcuiMl6ImkEB5na0TzOMyzrg8A/edit#


- Maxim Khutornenko


On Feb. 10, 2015, 12:49 a.m., lozh...@ebay.com zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30710/
> ---
> 
> (Updated Feb. 10, 2015, 12:49 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Bill Farner.
> 
> 
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> ## Problems
> 
> We are from eBay platform team. Previously, we used marathon to generate 
> Jenkins master instance in dedicated vms and recieve resource offer from same 
> dedicated vms. For the details, please refer to
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
> 
> Now, we found Aurora is more stable and powerful. We are moving from Marathon 
> to Aurora. During the move, we found there is no mesos role in Aurora now. 
> But we need use mesos role way to solve the problem in section "Frameworks 
> stopped receiving offers after a while" of the given url.
> 
> Here is a snippet of the problem description:
> 
> *We noticed occurred after we used Marathon to create the initial set of CI 
> masters. As those CI masters started registering themselves as frameworks, 
> Marathon stopped receiving any offers from Mesos; essentially, no new CI 
> masters could be launched. Let’s start with Marathon. In the DRF model, it 
> was unfair to treat Marathon in the same bucket/role alongside hundreds of 
> connected Jenkins frameworks. After launching all these Jenkins frameworks, 
> Marathon had a large resource share and Mesos would aggressively offer 
> resources to frameworks that were using little or no resources. Marathon was 
> placed last in priority and got starved out.*
> 
> *We decided to define a dedicated Mesos role for Marathon and to have all of 
> the Mesos slaves that were reserved for Jenkins master instances support that 
> Mesos role. Jenkins frameworks were left with the default role “*”.*This 
> solved the problem – Mesos offered resources per role and hence Marathon 
> never got starved out. A framework with a special role will get resource 
> offers from both slaves supporting that special role and also from the 
> default role “*”.* However, since we were using placement constraints, 
> Marathon accepted resource offers only from slaves that supported both the 
> role and the placement constraints.*
> ## Solution
> 
> So we add role feature is the source code to solve the problem in same way: 
> When accept a resource offer, Aurora will send back the needed resources to 
> Mesos with the mesos role in resource offer.
> 
> How to configure the Mesos role:
> 1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
> 
> We change the test cases according code change. Each changed test case is 
> green
> Merge https://github.com/zhanglong2015/incubator-aurora
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
> 1a158b4e0be94762ad0480e8ce74b19bacc90c97 
>   src/main/java/org/apache/aurora/scheduler

Re: Review Request 30710: add mesos role feature

[Zameer Manji]

> On Feb. 9, 2015, 4:56 p.m., Zameer Manji wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java,
> >  line 117
> > 
> >
> > I don't think we can safely do this right now. I know the default value 
> > is '*' 
> > (https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L128),
> >  but we have hit bugs in the past where Mesos will reject protobufs as 
> > changed when we start setting a previously unset field to a default value.
> > 
> > See https://issues.apache.org/jira/browse/MESOS-2309 for details.
> > 
> > Currently as this stands, I think this will cause existing Aurora 
> > instances to be rejected by the Master.
> 
> lozh...@ebay.com zhang wrote:
> How about if we left the role unset in register framework message if no 
> role specified in command line?

I think that's a good step to take. When you make this change can you reference 
MESOS-2309 as the reason why in a comment?


- Zameer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review71727
---


On Feb. 9, 2015, 4:49 p.m., lozh...@ebay.com zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30710/
> ---
> 
> (Updated Feb. 9, 2015, 4:49 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Bill Farner.
> 
> 
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> ## Problems
> 
> We are from eBay platform team. Previously, we used marathon to generate 
> Jenkins master instance in dedicated vms and recieve resource offer from same 
> dedicated vms. For the details, please refer to
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
> 
> Now, we found Aurora is more stable and powerful. We are moving from Marathon 
> to Aurora. During the move, we found there is no mesos role in Aurora now. 
> But we need use mesos role way to solve the problem in section "Frameworks 
> stopped receiving offers after a while" of the given url.
> 
> Here is a snippet of the problem description:
> 
> *We noticed occurred after we used Marathon to create the initial set of CI 
> masters. As those CI masters started registering themselves as frameworks, 
> Marathon stopped receiving any offers from Mesos; essentially, no new CI 
> masters could be launched. Let’s start with Marathon. In the DRF model, it 
> was unfair to treat Marathon in the same bucket/role alongside hundreds of 
> connected Jenkins frameworks. After launching all these Jenkins frameworks, 
> Marathon had a large resource share and Mesos would aggressively offer 
> resources to frameworks that were using little or no resources. Marathon was 
> placed last in priority and got starved out.*
> 
> *We decided to define a dedicated Mesos role for Marathon and to have all of 
> the Mesos slaves that were reserved for Jenkins master instances support that 
> Mesos role. Jenkins frameworks were left with the default role “*”.*This 
> solved the problem – Mesos offered resources per role and hence Marathon 
> never got starved out. A framework with a special role will get resource 
> offers from both slaves supporting that special role and also from the 
> default role “*”.* However, since we were using placement constraints, 
> Marathon accepted resource offers only from slaves that supported both the 
> role and the placement constraints.*
> ## Solution
> 
> So we add role feature is the source code to solve the problem in same way: 
> When accept a resource offer, Aurora will send back the needed resources to 
> Mesos with the mesos role in resource offer.
> 
> How to configure the Mesos role:
> 1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
> 
> We change the test cases according code change. Each changed test case is 
> green
> Merge https://github.com/zhanglong2015/incubator-aurora
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
> 1a158b4e0be94762ad0480e8ce74b19bacc90c97 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> 31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58 
>   src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 
> b5a3140e3560f790d1db496dca3c2ee0dc96a195 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  d0994203b5650f44ca2eb32e1e2aa61875163854 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> 5340d651b298ec8aa079e73d6d2f652fdf876293 
>   src/main/java/org/apache/aurora/sched

Re: Review Request 30710: add mesos role feature

[lozh...@ebay.com zhang]

> On Feb. 10, 2015, 12:56 a.m., Zameer Manji wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java,
> >  line 117
> > 
> >
> > I don't think we can safely do this right now. I know the default value 
> > is '*' 
> > (https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L128),
> >  but we have hit bugs in the past where Mesos will reject protobufs as 
> > changed when we start setting a previously unset field to a default value.
> > 
> > See https://issues.apache.org/jira/browse/MESOS-2309 for details.
> > 
> > Currently as this stands, I think this will cause existing Aurora 
> > instances to be rejected by the Master.

How about if we left the role unset in register framework message if no role 
specified in command line?


- lozh...@ebay.com


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review71727
---


On Feb. 10, 2015, 12:49 a.m., lozh...@ebay.com zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30710/
> ---
> 
> (Updated Feb. 10, 2015, 12:49 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Bill Farner.
> 
> 
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> ## Problems
> 
> We are from eBay platform team. Previously, we used marathon to generate 
> Jenkins master instance in dedicated vms and recieve resource offer from same 
> dedicated vms. For the details, please refer to
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
> 
> Now, we found Aurora is more stable and powerful. We are moving from Marathon 
> to Aurora. During the move, we found there is no mesos role in Aurora now. 
> But we need use mesos role way to solve the problem in section "Frameworks 
> stopped receiving offers after a while" of the given url.
> 
> Here is a snippet of the problem description:
> 
> *We noticed occurred after we used Marathon to create the initial set of CI 
> masters. As those CI masters started registering themselves as frameworks, 
> Marathon stopped receiving any offers from Mesos; essentially, no new CI 
> masters could be launched. Let’s start with Marathon. In the DRF model, it 
> was unfair to treat Marathon in the same bucket/role alongside hundreds of 
> connected Jenkins frameworks. After launching all these Jenkins frameworks, 
> Marathon had a large resource share and Mesos would aggressively offer 
> resources to frameworks that were using little or no resources. Marathon was 
> placed last in priority and got starved out.*
> 
> *We decided to define a dedicated Mesos role for Marathon and to have all of 
> the Mesos slaves that were reserved for Jenkins master instances support that 
> Mesos role. Jenkins frameworks were left with the default role “*”.*This 
> solved the problem – Mesos offered resources per role and hence Marathon 
> never got starved out. A framework with a special role will get resource 
> offers from both slaves supporting that special role and also from the 
> default role “*”.* However, since we were using placement constraints, 
> Marathon accepted resource offers only from slaves that supported both the 
> role and the placement constraints.*
> ## Solution
> 
> So we add role feature is the source code to solve the problem in same way: 
> When accept a resource offer, Aurora will send back the needed resources to 
> Mesos with the mesos role in resource offer.
> 
> How to configure the Mesos role:
> 1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
> 
> We change the test cases according code change. Each changed test case is 
> green
> Merge https://github.com/zhanglong2015/incubator-aurora
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
> 1a158b4e0be94762ad0480e8ce74b19bacc90c97 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> 31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58 
>   src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 
> b5a3140e3560f790d1db496dca3c2ee0dc96a195 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  d0994203b5650f44ca2eb32e1e2aa61875163854 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> 5340d651b298ec8aa079e73d6d2f652fdf876293 
>   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
> e1c29747c9854cf75bf63f6f085cf40ca68989af 
>   src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.ja

Re: Review Request 30710: add mesos role feature

[Zameer Manji]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review71727
---



src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java


I don't think we can safely do this right now. I know the default value is 
'*' 
(https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L128), 
but we have hit bugs in the past where Mesos will reject protobufs as changed 
when we start setting a previously unset field to a default value.

See https://issues.apache.org/jira/browse/MESOS-2309 for details.

Currently as this stands, I think this will cause existing Aurora instances 
to be rejected by the Master.


- Zameer Manji


On Feb. 9, 2015, 4:49 p.m., lozh...@ebay.com zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30710/
> ---
> 
> (Updated Feb. 9, 2015, 4:49 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Bill Farner.
> 
> 
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> ## Problems
> 
> We are from eBay platform team. Previously, we used marathon to generate 
> Jenkins master instance in dedicated vms and recieve resource offer from same 
> dedicated vms. For the details, please refer to
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
> 
> Now, we found Aurora is more stable and powerful. We are moving from Marathon 
> to Aurora. During the move, we found there is no mesos role in Aurora now. 
> But we need use mesos role way to solve the problem in section "Frameworks 
> stopped receiving offers after a while" of the given url.
> 
> Here is a snippet of the problem description:
> 
> *We noticed occurred after we used Marathon to create the initial set of CI 
> masters. As those CI masters started registering themselves as frameworks, 
> Marathon stopped receiving any offers from Mesos; essentially, no new CI 
> masters could be launched. Let’s start with Marathon. In the DRF model, it 
> was unfair to treat Marathon in the same bucket/role alongside hundreds of 
> connected Jenkins frameworks. After launching all these Jenkins frameworks, 
> Marathon had a large resource share and Mesos would aggressively offer 
> resources to frameworks that were using little or no resources. Marathon was 
> placed last in priority and got starved out.*
> 
> *We decided to define a dedicated Mesos role for Marathon and to have all of 
> the Mesos slaves that were reserved for Jenkins master instances support that 
> Mesos role. Jenkins frameworks were left with the default role “*”.*This 
> solved the problem – Mesos offered resources per role and hence Marathon 
> never got starved out. A framework with a special role will get resource 
> offers from both slaves supporting that special role and also from the 
> default role “*”.* However, since we were using placement constraints, 
> Marathon accepted resource offers only from slaves that supported both the 
> role and the placement constraints.*
> ## Solution
> 
> So we add role feature is the source code to solve the problem in same way: 
> When accept a resource offer, Aurora will send back the needed resources to 
> Mesos with the mesos role in resource offer.
> 
> How to configure the Mesos role:
> 1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
> 
> We change the test cases according code change. Each changed test case is 
> green
> Merge https://github.com/zhanglong2015/incubator-aurora
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
> 1a158b4e0be94762ad0480e8ce74b19bacc90c97 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> 31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58 
>   src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 
> b5a3140e3560f790d1db496dca3c2ee0dc96a195 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  d0994203b5650f44ca2eb32e1e2aa61875163854 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> 5340d651b298ec8aa079e73d6d2f652fdf876293 
>   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
> e1c29747c9854cf75bf63f6f085cf40ca68989af 
>   src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java 
> 422d5a9a42310979752eb7282658316c2b772419 
>   src/test/java/org/apache/aurora/scheduler/configuration/ResourcesTest.java 
> d6febb8998e05257cabe8d193cefa0b6c79f197e 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.jav

Re: Review Request 30710: add mesos role feature

[lozh...@ebay.com zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/
---

(Updated Feb. 9, 2015, 4:01 a.m.)


Review request for Aurora and Bill Farner.


Changes
---

add link to jira


Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109


Repository: aurora


Description
---

## Problems

We are from eBay platform team. Previously, we used marathon to generate 
Jenkins master instance in dedicated vms and recieve resource offer from same 
dedicated vms. For the details, please refer to
http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU

Now, we found Aurora is more stable and powerful. We are moving from Marathon 
to Aurora. During the move, we found there is no mesos role in Aurora now. But 
we need use mesos role way to solve the problem in section "Frameworks stopped 
receiving offers after a while" of the given url.

Here is a snippet of the problem description:

*We noticed occurred after we used Marathon to create the initial set of CI 
masters. As those CI masters started registering themselves as frameworks, 
Marathon stopped receiving any offers from Mesos; essentially, no new CI 
masters could be launched. Let’s start with Marathon. In the DRF model, it was 
unfair to treat Marathon in the same bucket/role alongside hundreds of 
connected Jenkins frameworks. After launching all these Jenkins frameworks, 
Marathon had a large resource share and Mesos would aggressively offer 
resources to frameworks that were using little or no resources. Marathon was 
placed last in priority and got starved out.*

*We decided to define a dedicated Mesos role for Marathon and to have all of 
the Mesos slaves that were reserved for Jenkins master instances support that 
Mesos role. Jenkins frameworks were left with the default role “*”.*This solved 
the problem – Mesos offered resources per role and hence Marathon never got 
starved out. A framework with a special role will get resource offers from both 
slaves supporting that special role and also from the default role “*”.* 
However, since we were using placement constraints, Marathon accepted resource 
offers only from slaves that supported both the role and the placement 
constraints.*
## Solution

So we add role feature is the source code to solve the problem in same way: 
When accept a resource offer, Aurora will send back the needed resources to 
Mesos with the mesos role in resource offer.

How to configure the Mesos role:
1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.

We change the test cases according code change. Each changed test case is green
Merge https://github.com/zhanglong2015/incubator-aurora


Diffs
-

  src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
1a158b4e0be94762ad0480e8ce74b19bacc90c97 
  src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58 
  src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 
b5a3140e3560f790d1db496dca3c2ee0dc96a195 
  
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
 d0994203b5650f44ca2eb32e1e2aa61875163854 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
5340d651b298ec8aa079e73d6d2f652fdf876293 
  src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
e1c29747c9854cf75bf63f6f085cf40ca68989af 
  src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java 
422d5a9a42310979752eb7282658316c2b772419 
  src/test/java/org/apache/aurora/scheduler/configuration/ResourcesTest.java 
d6febb8998e05257cabe8d193cefa0b6c79f197e 
  src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
5f08d00d39f016af9bc296e517ad49b66ab5a8de 
  src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
411a55a8d85f60bb2703468f2d69b64b2736eee4 

Diff: https://reviews.apache.org/r/30710/diff/


Testing
---

:buildSrc:compileJava UP-TO-DATE
:buildSrc:compileGroovy UP-TO-DATE
:buildSrc:processResources UP-TO-DATE
:buildSrc:classes UP-TO-DATE
:buildSrc:jar UP-TO-DATE
:buildSrc:assemble UP-TO-DATE
:buildSrc:compileTestJava UP-TO-DATE
:buildSrc:compileTestGroovy UP-TO-DATE
:buildSrc:processTestResources UP-TO-DATE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test UP-TO-DATE
:buildSrc:check UP-TO-DATE
:buildSrc:build UP-TO-DATE
:api:generateThriftJava
:api:classesThrift
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
:api:checkPython
:api:generateThriftEntitiesJava
:api:classesThriftEntities
:api:compileJava UP-TO-DATE
:api:generateThriftResources
:api:processResources UP-TO-DATE
:api:classes
:api:jar
:compileJava
Note: Writing 
file:/root/incubator-aurora/dist/classes/main/com/twitter/common/args/apt/cmdline.arg.inf

Re: Review Request 30710: add mesos role feature

[Aurora ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review71407
---

Ship it!


Master (5059040) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Feb. 6, 2015, 9:23 a.m., lozh...@ebay.com zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30710/
> ---
> 
> (Updated Feb. 6, 2015, 9:23 a.m.)
> 
> 
> Review request for Aurora and Bill Farner.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> ## Problems
> 
> We are from eBay platform team. Previously, we used marathon to generate 
> Jenkins master instance in dedicated vms and recieve resource offer from same 
> dedicated vms. For the details, please refer to
> http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
> 
> Now, we found Aurora is more stable and powerful. We are moving from Marathon 
> to Aurora. During the move, we found there is no mesos role in Aurora now. 
> But we need use mesos role way to solve the problem in section "Frameworks 
> stopped receiving offers after a while" of the given url.
> 
> Here is a snippet of the problem description:
> 
> *We noticed occurred after we used Marathon to create the initial set of CI 
> masters. As those CI masters started registering themselves as frameworks, 
> Marathon stopped receiving any offers from Mesos; essentially, no new CI 
> masters could be launched. Let’s start with Marathon. In the DRF model, it 
> was unfair to treat Marathon in the same bucket/role alongside hundreds of 
> connected Jenkins frameworks. After launching all these Jenkins frameworks, 
> Marathon had a large resource share and Mesos would aggressively offer 
> resources to frameworks that were using little or no resources. Marathon was 
> placed last in priority and got starved out.*
> 
> *We decided to define a dedicated Mesos role for Marathon and to have all of 
> the Mesos slaves that were reserved for Jenkins master instances support that 
> Mesos role. Jenkins frameworks were left with the default role “*”.*This 
> solved the problem – Mesos offered resources per role and hence Marathon 
> never got starved out. A framework with a special role will get resource 
> offers from both slaves supporting that special role and also from the 
> default role “*”.* However, since we were using placement constraints, 
> Marathon accepted resource offers only from slaves that supported both the 
> role and the placement constraints.*
> ## Solution
> 
> So we add role feature is the source code to solve the problem in same way: 
> When accept a resource offer, Aurora will send back the needed resources to 
> Mesos with the mesos role in resource offer.
> 
> How to configure the Mesos role:
> 1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
> 
> We change the test cases according code change. Each changed test case is 
> green
> Merge https://github.com/zhanglong2015/incubator-aurora
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
> 1a158b4e0be94762ad0480e8ce74b19bacc90c97 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> 31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58 
>   src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 
> b5a3140e3560f790d1db496dca3c2ee0dc96a195 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  d0994203b5650f44ca2eb32e1e2aa61875163854 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> 5340d651b298ec8aa079e73d6d2f652fdf876293 
>   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
> e1c29747c9854cf75bf63f6f085cf40ca68989af 
>   src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java 
> 422d5a9a42310979752eb7282658316c2b772419 
>   src/test/java/org/apache/aurora/scheduler/configuration/ResourcesTest.java 
> d6febb8998e05257cabe8d193cefa0b6c79f197e 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
> 5f08d00d39f016af9bc296e517ad49b66ab5a8de 
>   src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
> 411a55a8d85f60bb2703468f2d69b64b2736eee4 
> 
> Diff: https://reviews.apache.org/r/30710/diff/
> 
> 
> Testing
> ---
> 
> :buildSrc:compileJava UP-TO-DATE
> :buildSrc:compileGroovy UP-TO-DATE
> :buildSrc:processResources UP-TO-DATE
> :buildSrc:classes UP-TO-DATE
> :buildSrc:jar UP-TO-DATE
> :buildSrc:assemble UP-TO-DATE
> :buildSrc:compileTestJava UP-TO-DATE
> :buildSrc:compileTestGroovy UP-TO-DATE
> :buildSrc:processTestRes

Re: Review Request 30710: add mesos role feature

[lozh...@ebay.com zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/
---

(Updated Feb. 6, 2015, 9:23 a.m.)


Review request for Aurora and Bill Farner.


Changes
---

align code style with apache


Repository: aurora


Description
---

## Problems

We are from eBay platform team. Previously, we used marathon to generate 
Jenkins master instance in dedicated vms and recieve resource offer from same 
dedicated vms. For the details, please refer to
http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU

Now, we found Aurora is more stable and powerful. We are moving from Marathon 
to Aurora. During the move, we found there is no mesos role in Aurora now. But 
we need use mesos role way to solve the problem in section "Frameworks stopped 
receiving offers after a while" of the given url.

Here is a snippet of the problem description:

*We noticed occurred after we used Marathon to create the initial set of CI 
masters. As those CI masters started registering themselves as frameworks, 
Marathon stopped receiving any offers from Mesos; essentially, no new CI 
masters could be launched. Let’s start with Marathon. In the DRF model, it was 
unfair to treat Marathon in the same bucket/role alongside hundreds of 
connected Jenkins frameworks. After launching all these Jenkins frameworks, 
Marathon had a large resource share and Mesos would aggressively offer 
resources to frameworks that were using little or no resources. Marathon was 
placed last in priority and got starved out.*

*We decided to define a dedicated Mesos role for Marathon and to have all of 
the Mesos slaves that were reserved for Jenkins master instances support that 
Mesos role. Jenkins frameworks were left with the default role “*”.*This solved 
the problem – Mesos offered resources per role and hence Marathon never got 
starved out. A framework with a special role will get resource offers from both 
slaves supporting that special role and also from the default role “*”.* 
However, since we were using placement constraints, Marathon accepted resource 
offers only from slaves that supported both the role and the placement 
constraints.*
## Solution

So we add role feature is the source code to solve the problem in same way: 
When accept a resource offer, Aurora will send back the needed resources to 
Mesos with the mesos role in resource offer.

How to configure the Mesos role:
1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.

We change the test cases according code change. Each changed test case is green
Merge https://github.com/zhanglong2015/incubator-aurora


Diffs (updated)
-

  src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
1a158b4e0be94762ad0480e8ce74b19bacc90c97 
  src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58 
  src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 
b5a3140e3560f790d1db496dca3c2ee0dc96a195 
  
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
 d0994203b5650f44ca2eb32e1e2aa61875163854 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
5340d651b298ec8aa079e73d6d2f652fdf876293 
  src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
e1c29747c9854cf75bf63f6f085cf40ca68989af 
  src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java 
422d5a9a42310979752eb7282658316c2b772419 
  src/test/java/org/apache/aurora/scheduler/configuration/ResourcesTest.java 
d6febb8998e05257cabe8d193cefa0b6c79f197e 
  src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
5f08d00d39f016af9bc296e517ad49b66ab5a8de 
  src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
411a55a8d85f60bb2703468f2d69b64b2736eee4 

Diff: https://reviews.apache.org/r/30710/diff/


Testing
---

:buildSrc:compileJava UP-TO-DATE
:buildSrc:compileGroovy UP-TO-DATE
:buildSrc:processResources UP-TO-DATE
:buildSrc:classes UP-TO-DATE
:buildSrc:jar UP-TO-DATE
:buildSrc:assemble UP-TO-DATE
:buildSrc:compileTestJava UP-TO-DATE
:buildSrc:compileTestGroovy UP-TO-DATE
:buildSrc:processTestResources UP-TO-DATE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test UP-TO-DATE
:buildSrc:check UP-TO-DATE
:buildSrc:build UP-TO-DATE
:api:generateThriftJava
:api:classesThrift
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
:api:checkPython
:api:generateThriftEntitiesJava
:api:classesThriftEntities
:api:compileJava UP-TO-DATE
:api:generateThriftResources
:api:processResources UP-TO-DATE
:api:classes
:api:jar
:compileJava
Note: Writing 
file:/root/incubator-aurora/dist/classes/main/com/twitter/common/args/apt/cmdline.arg.info.txt.2
:processResources
:classes
:jar
:assemble
:c

Re: Review Request 30710: add mesos role feature

[Bill Farner]

> On Feb. 6, 2015, 7:02 a.m., Aurora ReviewBot wrote:
> > Master (5059040) is red with this patch.
> >   ./build-support/jenkins/build.sh
> > 
> > 1 warning
> > 
> > :processJmhResources UP-TO-DATE
> > :jmhClasses
> > :checkstyleJmh
> > :jsHint
> > :checkstyleMain[ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java:18:
> >  Wrong order for 'java.util.Collections' import. Order should be: java, 
> > javax, scala, com, net, org. Each group should be separated by a 
> > single blank line.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java:51:
> >  Wrong order for 'com.google.common.annotations.VisibleForTesting' import. 
> > Order should be: java, javax, scala, com, net, org. Each group 
> > should be separated by a single blank line.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java:185:
> >  Line is longer than 100 characters (found 111).
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java:188:5:
> >  File contains tab characters (this is the first instance).
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:103:
> >  Line has trailing spaces.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:105:1:
> >  File contains tab characters (this is the first instance).
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:105:5:
> >  'if' is not followed by whitespace.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:128:83:
> >  ',' is not followed by whitespace.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:145:
> >  Line has trailing spaces.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:317:
> >  Line has trailing spaces.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:319:
> >  Line has trailing spaces.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:331:
> >  Line has trailing spaces.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java:56:
> >  Line has trailing spaces.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java:115:27:
> >  '+' is not preceded with whitespace.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java:118:1:
> >  File contains tab characters (this is the first instance).
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java:18:
> >  Wrong order for 'java.util.List' import. Order should be: java, javax, 
> > scala, com, net, org. Each group should be separated by a single 
> > blank line.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java:47:
> >  Wrong order for 'com.google.common.annotations.VisibleForTesting' import. 
> > Order should be: java, javax, scala, com, net, org. Each group 
> > should be separated by a single blank line.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java:115:
> >  File contains a sequence of empty lines.
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java:120:1:
> >  File contains tab characters (this is the first instance).
> > [ant:checkstyle] 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java:177:
> >  Line is longer than 100 characters (found 104).
> >  FAILED
> > 
> > FAILURE: Build failed with an exception.
> > 
> > * What went wrong:
> > Execution failed for task ':checkstyleMain'.
> > > Checkstyle rule violations were found. See the report at: 
> > > f

Re: Review Request 30710: add mesos role feature

[Aurora ReviewBot]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review71386
---


Master (5059040) is red with this patch.
  ./build-support/jenkins/build.sh

1 warning

:processJmhResources UP-TO-DATE
:jmhClasses
:checkstyleJmh
:jsHint
:checkstyleMain[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java:18:
 Wrong order for 'java.util.Collections' import. Order should be: java, javax, 
scala, com, net, org. Each group should be separated by a single blank 
line.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java:51:
 Wrong order for 'com.google.common.annotations.VisibleForTesting' import. 
Order should be: java, javax, scala, com, net, org. Each group should 
be separated by a single blank line.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java:185:
 Line is longer than 100 characters (found 111).
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java:188:5:
 File contains tab characters (this is the first instance).
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:103:
 Line has trailing spaces.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:105:1:
 File contains tab characters (this is the first instance).
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:105:5:
 'if' is not followed by whitespace.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:128:83:
 ',' is not followed by whitespace.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:145:
 Line has trailing spaces.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:317:
 Line has trailing spaces.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:319:
 Line has trailing spaces.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/configuration/Resources.java:331:
 Line has trailing spaces.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java:56:
 Line has trailing spaces.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java:115:27:
 '+' is not preceded with whitespace.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java:118:1:
 File contains tab characters (this is the first instance).
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java:18:
 Wrong order for 'java.util.List' import. Order should be: java, javax, scala, 
com, net, org. Each group should be separated by a single blank line.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java:47:
 Wrong order for 'com.google.common.annotations.VisibleForTesting' import. 
Order should be: java, javax, scala, com, net, org. Each group should 
be separated by a single blank line.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java:115:
 File contains a sequence of empty lines.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java:120:1:
 File contains tab characters (this is the first instance).
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java:177:
 Line is longer than 100 characters (found 104).
 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':checkstyleMain'.
> Checkstyle rule violations were found. See the report at: 
> file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/checkstyle/main.xml

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or

Review Request 30710: add mesos role feature

[lozh...@ebay.com zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/
---

Review request for Aurora and Bill Farner.


Repository: aurora


Description
---

## Problems

We are from eBay platform team. Previously, we used marathon to generate 
Jenkins master instance in dedicated vms and recieve resource offer from same 
dedicated vms. For the details, please refer to
http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU

Now, we found Aurora is more stable and powerful. We are moving from Marathon 
to Aurora. During the move, we found there is no mesos role in Aurora now. But 
we need use mesos role way to solve the problem in section "Frameworks stopped 
receiving offers after a while" of the given url.

Here is a snippet of the problem description:

*We noticed occurred after we used Marathon to create the initial set of CI 
masters. As those CI masters started registering themselves as frameworks, 
Marathon stopped receiving any offers from Mesos; essentially, no new CI 
masters could be launched. Let’s start with Marathon. In the DRF model, it was 
unfair to treat Marathon in the same bucket/role alongside hundreds of 
connected Jenkins frameworks. After launching all these Jenkins frameworks, 
Marathon had a large resource share and Mesos would aggressively offer 
resources to frameworks that were using little or no resources. Marathon was 
placed last in priority and got starved out.*

*We decided to define a dedicated Mesos role for Marathon and to have all of 
the Mesos slaves that were reserved for Jenkins master instances support that 
Mesos role. Jenkins frameworks were left with the default role “*”.*This solved 
the problem – Mesos offered resources per role and hence Marathon never got 
starved out. A framework with a special role will get resource offers from both 
slaves supporting that special role and also from the default role “*”.* 
However, since we were using placement constraints, Marathon accepted resource 
offers only from slaves that supported both the role and the placement 
constraints.*
## Solution

So we add role feature is the source code to solve the problem in same way: 
When accept a resource offer, Aurora will send back the needed resources to 
Mesos with the mesos role in resource offer.

How to configure the Mesos role:
1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.

We change the test cases according code change. Each changed test case is green
Merge https://github.com/zhanglong2015/incubator-aurora


Diffs
-

  src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 
1a158b4e0be94762ad0480e8ce74b19bacc90c97 
  src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58 
  src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 
b5a3140e3560f790d1db496dca3c2ee0dc96a195 
  
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
 d0994203b5650f44ca2eb32e1e2aa61875163854 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
5340d651b298ec8aa079e73d6d2f652fdf876293 
  src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
e1c29747c9854cf75bf63f6f085cf40ca68989af 
  src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java 
422d5a9a42310979752eb7282658316c2b772419 
  src/test/java/org/apache/aurora/scheduler/configuration/ResourcesTest.java 
d6febb8998e05257cabe8d193cefa0b6c79f197e 
  src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
5f08d00d39f016af9bc296e517ad49b66ab5a8de 
  src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
411a55a8d85f60bb2703468f2d69b64b2736eee4 

Diff: https://reviews.apache.org/r/30710/diff/


Testing
---

:buildSrc:compileJava UP-TO-DATE
:buildSrc:compileGroovy UP-TO-DATE
:buildSrc:processResources UP-TO-DATE
:buildSrc:classes UP-TO-DATE
:buildSrc:jar UP-TO-DATE
:buildSrc:assemble UP-TO-DATE
:buildSrc:compileTestJava UP-TO-DATE
:buildSrc:compileTestGroovy UP-TO-DATE
:buildSrc:processTestResources UP-TO-DATE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test UP-TO-DATE
:buildSrc:check UP-TO-DATE
:buildSrc:build UP-TO-DATE
:api:generateThriftJava
:api:classesThrift
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
:api:checkPython
:api:generateThriftEntitiesJava
:api:classesThriftEntities
:api:compileJava UP-TO-DATE
:api:generateThriftResources
:api:processResources UP-TO-DATE
:api:classes
:api:jar
:compileJava
Note: Writing 
file:/root/incubator-aurora/dist/classes/main/com/twitter/common/args/apt/cmdline.arg.info.txt.2
:processResources
:classes
:jar
:assemble
:compileJmhJava
warning: Supported source version 'RELEASE_6' from annotation processor 
'org.op