date:20160503

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

2016-05-03 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46835/#review131609
---



This patch does not apply cleanly against master (8a2fc4c), do you need to 
rebase?

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On May 4, 2016, 1:45 a.m., Joshua Cohen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46835/
> ---
> 
> (Updated May 4, 2016, 1:45 a.m.)
> 
> 
> Review request for Aurora, John Sirois, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1636, AURORA-1637, AURORA-1638, and AURORA-1639
> https://issues.apache.org/jira/browse/AURORA-1636
> https://issues.apache.org/jira/browse/AURORA-1637
> https://issues.apache.org/jira/browse/AURORA-1638
> https://issues.apache.org/jira/browse/AURORA-1639
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> A few notes:
> 
> 1. It's not possible to configure Mesos 0.27.x to launch docker tasks due to 
> a bug in parsing the docker_store_dir flag. Fixed here: 
> https://reviews.apache.org/r/43451/ but has not been backported to Mesos 
> 0.27. This means we can only launch tasks that use AppC images until we 
> upgrade our Mesos dependency to 0.28.x. The good news is I've confirmed that 
> launching tasks with Docker images *does* work by using Aurora linked against 
> 0.27.x but running Mesos 0.28.x in Vagrant.
> 1. In order to work around the setuid issues (i.e. task is launched as root, 
> but the executor cannot setuid because the role-user does not exist), I've 
> mounted /etc/passwd and /etc/group into the container and added a new flag, 
> `thermos_run_as_job_role`, to the scheduler. This flag is only used when 
> launching a task with a filesystem image, and causes us to add 
> `--execute-as-user ` to the thermos executor commandline.
> 1. The Mesos unified containerizer does not automatically create mount points 
> in the filesystem from the image. It expects the full path to the mount to 
> exist in the image. For /etc/passwd and /etc/groups this is not a problem, 
> but for the announcer acls file it was. I ended up moving the announcer acl 
> file into its own directory and mount that instead. In conjunction with this 
> I also had to modify our http_example Dockerfile to explicitly create that 
> mount point. A case could be made for sticking with the current path and just 
> creating an empty file in the image, I felt that creating an empty directory 
> was slightly less gross. This is tracked by 
> https://issues.apache.org/jira/browse/MESOS-5229.
> 1. The AppC image for end to end tests is created by running 
> [docker2aci](https://github.com/appc/docker2aci) on our http_example docker 
> image. The base box needed to be upgraded to add this utility. I haven't 
> published the new base box yet even though I've updated the Vagrantfile to 
> point to version 6. Once this review has been approved and I'm sure there's 
> no further changes that need to be made I'll publish the base box before 
> committing.
> 
> 
> Diffs
> -
> 
>   3rdparty/python/requirements.txt 666c4ae487332f01380cfce76f0d97e2c6049c8e 
>   RELEASE-NOTES.md 7a37d0d69f688bece624628fe5b98efc85d506a2 
>   Vagrantfile 3f126ee348d0f95d6f159b62280de79f41e87e2e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 38470951e4482753fcada109ab12546a2fb146ce 
>   build-support/packer/build.sh 76197c31c365aa3d8a67049da40b2976c1e25d22 
>   docs/reference/configuration.md 9fcfdfcd9ab793e888ca2bba2035d5122142a5ab 
>   docs/reference/scheduler-configuration.md 
> d2262f79edfde23eccd87bae7f1cf319b63b1103 
>   examples/vagrant/announcer-auth.json  
>   examples/vagrant/mesos_config/etc_mesos-slave/appc_store_dir PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/image_providers PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/image_provisioner_backend 
> PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/isolation PRE-CREATION 
>   examples/vagrant/upstart/aurora-scheduler.conf 
> 084016abc169ed82b7ed00f5d14aea2e0ff38a49 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
>  9a15a4b5c1cdc3efbf0588e8bd81636b66e938ef 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
>  32f2fa90b21189180e2bcd65a3cebf13f6551646 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> b325106c7f45b1ad1657221aaa39e3a428719ab0 
>   src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java 
> 9aadcebf547bd1eb4b4e238507e27ae2b699f473 
>   src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 
> 9eadf70240738030593a819a4a248588c9d1c3

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

2016-05-03 Thread Joshua Cohen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46835/
---

(Updated May 4, 2016, 1:45 a.m.)


Review request for Aurora, John Sirois, Maxim Khutornenko, and Bill Farner.


Changes
---

Review feedback and checkstyle issues.


Bugs: AURORA-1636, AURORA-1637, AURORA-1638, and AURORA-1639
https://issues.apache.org/jira/browse/AURORA-1636
https://issues.apache.org/jira/browse/AURORA-1637
https://issues.apache.org/jira/browse/AURORA-1638
https://issues.apache.org/jira/browse/AURORA-1639


Repository: aurora


Description
---

A few notes:

1. It's not possible to configure Mesos 0.27.x to launch docker tasks due to a 
bug in parsing the docker_store_dir flag. Fixed here: 
https://reviews.apache.org/r/43451/ but has not been backported to Mesos 0.27. 
This means we can only launch tasks that use AppC images until we upgrade our 
Mesos dependency to 0.28.x. The good news is I've confirmed that launching 
tasks with Docker images *does* work by using Aurora linked against 0.27.x but 
running Mesos 0.28.x in Vagrant.
1. In order to work around the setuid issues (i.e. task is launched as root, 
but the executor cannot setuid because the role-user does not exist), I've 
mounted /etc/passwd and /etc/group into the container and added a new flag, 
`thermos_run_as_job_role`, to the scheduler. This flag is only used when 
launching a task with a filesystem image, and causes us to add 
`--execute-as-user ` to the thermos executor commandline.
1. The Mesos unified containerizer does not automatically create mount points 
in the filesystem from the image. It expects the full path to the mount to 
exist in the image. For /etc/passwd and /etc/groups this is not a problem, but 
for the announcer acls file it was. I ended up moving the announcer acl file 
into its own directory and mount that instead. In conjunction with this I also 
had to modify our http_example Dockerfile to explicitly create that mount 
point. A case could be made for sticking with the current path and just 
creating an empty file in the image, I felt that creating an empty directory 
was slightly less gross. This is tracked by 
https://issues.apache.org/jira/browse/MESOS-5229.
1. The AppC image for end to end tests is created by running 
[docker2aci](https://github.com/appc/docker2aci) on our http_example docker 
image. The base box needed to be upgraded to add this utility. I haven't 
published the new base box yet even though I've updated the Vagrantfile to 
point to version 6. Once this review has been approved and I'm sure there's no 
further changes that need to be made I'll publish the base box before 
committing.


Diffs (updated)
-

  3rdparty/python/requirements.txt 666c4ae487332f01380cfce76f0d97e2c6049c8e 
  RELEASE-NOTES.md 7a37d0d69f688bece624628fe5b98efc85d506a2 
  Vagrantfile 3f126ee348d0f95d6f159b62280de79f41e87e2e 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
38470951e4482753fcada109ab12546a2fb146ce 
  build-support/packer/build.sh 76197c31c365aa3d8a67049da40b2976c1e25d22 
  docs/reference/configuration.md 9fcfdfcd9ab793e888ca2bba2035d5122142a5ab 
  docs/reference/scheduler-configuration.md 
d2262f79edfde23eccd87bae7f1cf319b63b1103 
  examples/vagrant/announcer-auth.json  
  examples/vagrant/mesos_config/etc_mesos-slave/appc_store_dir PRE-CREATION 
  examples/vagrant/mesos_config/etc_mesos-slave/image_providers PRE-CREATION 
  examples/vagrant/mesos_config/etc_mesos-slave/image_provisioner_backend 
PRE-CREATION 
  examples/vagrant/mesos_config/etc_mesos-slave/isolation PRE-CREATION 
  examples/vagrant/upstart/aurora-scheduler.conf 
084016abc169ed82b7ed00f5d14aea2e0ff38a49 
  
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
 9a15a4b5c1cdc3efbf0588e8bd81636b66e938ef 
  
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
 32f2fa90b21189180e2bcd65a3cebf13f6551646 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
b325106c7f45b1ad1657221aaa39e3a428719ab0 
  src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java 
9aadcebf547bd1eb4b4e238507e27ae2b699f473 
  src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 
9eadf70240738030593a819a4a248588c9d1c35b 
  src/main/java/org/apache/aurora/scheduler/storage/db/views/DbContainer.java 
ae97638fa544dd8f8afbaa19b1dd31f5a1dc43d8 
  src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java 
a7523c4b258030bcfb2e457b083242ffa865a98a 
  src/main/python/apache/aurora/config/schema/base.py 
00be8747d70dbf1cb370f09536588f8602d8fcce 
  src/main/python/apache/aurora/config/thrift.py 
928ca9313b2c2062a322ba80b504a09c55e5377f 
  src/main/python/apache/aurora/executor/common/sandbox.py 
36f1eabedc3ae47b23d9ab2ac0ab7a576ea36fd7 
  
src/main/resources/org/apache/aurora/schedule

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

2016-05-03 Thread Joshua Cohen



> On May 3, 2016, 11 p.m., Stephan Erb wrote:
> > src/main/python/apache/aurora/config/schema/base.py, line 146
> > 
> >
> > This can be removed, I guess.

Yep, missed that, thanks.


- Joshua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46835/#review131577
---


On May 3, 2016, 10:04 p.m., Joshua Cohen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46835/
> ---
> 
> (Updated May 3, 2016, 10:04 p.m.)
> 
> 
> Review request for Aurora, John Sirois, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1636, AURORA-1637, AURORA-1638, and AURORA-1639
> https://issues.apache.org/jira/browse/AURORA-1636
> https://issues.apache.org/jira/browse/AURORA-1637
> https://issues.apache.org/jira/browse/AURORA-1638
> https://issues.apache.org/jira/browse/AURORA-1639
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> A few notes:
> 
> 1. It's not possible to configure Mesos 0.27.x to launch docker tasks due to 
> a bug in parsing the docker_store_dir flag. Fixed here: 
> https://reviews.apache.org/r/43451/ but has not been backported to Mesos 
> 0.27. This means we can only launch tasks that use AppC images until we 
> upgrade our Mesos dependency to 0.28.x. The good news is I've confirmed that 
> launching tasks with Docker images *does* work by using Aurora linked against 
> 0.27.x but running Mesos 0.28.x in Vagrant.
> 1. In order to work around the setuid issues (i.e. task is launched as root, 
> but the executor cannot setuid because the role-user does not exist), I've 
> mounted /etc/passwd and /etc/group into the container and added a new flag, 
> `thermos_run_as_job_role`, to the scheduler. This flag is only used when 
> launching a task with a filesystem image, and causes us to add 
> `--execute-as-user ` to the thermos executor commandline.
> 1. The Mesos unified containerizer does not automatically create mount points 
> in the filesystem from the image. It expects the full path to the mount to 
> exist in the image. For /etc/passwd and /etc/groups this is not a problem, 
> but for the announcer acls file it was. I ended up moving the announcer acl 
> file into its own directory and mount that instead. In conjunction with this 
> I also had to modify our http_example Dockerfile to explicitly create that 
> mount point. A case could be made for sticking with the current path and just 
> creating an empty file in the image, I felt that creating an empty directory 
> was slightly less gross. This is tracked by 
> https://issues.apache.org/jira/browse/MESOS-5229.
> 1. The AppC image for end to end tests is created by running 
> [docker2aci](https://github.com/appc/docker2aci) on our http_example docker 
> image. The base box needed to be upgraded to add this utility. I haven't 
> published the new base box yet even though I've updated the Vagrantfile to 
> point to version 6. Once this review has been approved and I'm sure there's 
> no further changes that need to be made I'll publish the base box before 
> committing.
> 
> 
> Diffs
> -
> 
>   3rdparty/python/requirements.txt 666c4ae487332f01380cfce76f0d97e2c6049c8e 
>   RELEASE-NOTES.md 7a37d0d69f688bece624628fe5b98efc85d506a2 
>   Vagrantfile 3f126ee348d0f95d6f159b62280de79f41e87e2e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 38470951e4482753fcada109ab12546a2fb146ce 
>   build-support/packer/build.sh 76197c31c365aa3d8a67049da40b2976c1e25d22 
>   docs/reference/configuration.md 9fcfdfcd9ab793e888ca2bba2035d5122142a5ab 
>   docs/reference/scheduler-configuration.md 
> d2262f79edfde23eccd87bae7f1cf319b63b1103 
>   examples/vagrant/announcer-auth.json  
>   examples/vagrant/mesos_config/etc_mesos-slave/appc_store_dir PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/image_providers PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/image_provisioner_backend 
> PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/isolation PRE-CREATION 
>   examples/vagrant/upstart/aurora-scheduler.conf 
> 084016abc169ed82b7ed00f5d14aea2e0ff38a49 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
>  9a15a4b5c1cdc3efbf0588e8bd81636b66e938ef 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
>  32f2fa90b21189180e2bcd65a3cebf13f6551646 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> b325106c7f45b1ad1657221aaa39e3a428719ab0 
>   src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java 
> 9aadcebf547bd1eb4b4e238507e27ae2b699f473 
>   src/main/java/org/apache/auror

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

2016-05-03 Thread Joshua Cohen



> On May 4, 2016, 12:19 a.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java,
> >  lines 76-78
> > 
> >
> > Does this have to be bundled into this RB? We are still using this path 
> > in DEFAULT_CHECKPOINT_ROOT on the thermos side. Perhaps, have a more 
> > thorough cleanup in a separate patch?

I need to drop the mount for the e2e test to work. I can drop the mount and 
leave the flag around unused for a later cleanup, but that will require 
checkstyle suppressions. If you'd prefer I can send out a separate patch that 
drops everything related to this arg and rebase this on top of it once it lands.


> On May 4, 2016, 12:19 a.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java,
> >  line 84
> > 
> >
> > Shouldn't this and `DBImage` go into `DBContainer` now that 
> > `TaskConfig` does not have an `image` property?

I came up with something workable. Since there's no concept of a mesos 
container in the database (and I don't think it makes sense to add a DB table 
just so we can have a symmetrical mapping), I'm just mapping the image directly 
to DbContainer. This is slightly different than the case for Docker, where we 
map a DockerContainer. 

I'm not sure it's necessarily better though. Let me know what you think ;).


> On May 4, 2016, 12:19 a.m., Maxim Khutornenko wrote:
> > src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java,
> >  line 432
> > 
> >
> > Wouldn't a fake imageId suffice here?

Sure, will change.


> On May 4, 2016, 12:19 a.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line 
> > 201
> > 
> >
> > This name is no longer accurate given that we actually use 
> > ContainerInfo.Builder inside.

I just moved this code directly inside the calling method.


- Joshua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46835/#review131585
---


On May 3, 2016, 10:04 p.m., Joshua Cohen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46835/
> ---
> 
> (Updated May 3, 2016, 10:04 p.m.)
> 
> 
> Review request for Aurora, John Sirois, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1636, AURORA-1637, AURORA-1638, and AURORA-1639
> https://issues.apache.org/jira/browse/AURORA-1636
> https://issues.apache.org/jira/browse/AURORA-1637
> https://issues.apache.org/jira/browse/AURORA-1638
> https://issues.apache.org/jira/browse/AURORA-1639
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> A few notes:
> 
> 1. It's not possible to configure Mesos 0.27.x to launch docker tasks due to 
> a bug in parsing the docker_store_dir flag. Fixed here: 
> https://reviews.apache.org/r/43451/ but has not been backported to Mesos 
> 0.27. This means we can only launch tasks that use AppC images until we 
> upgrade our Mesos dependency to 0.28.x. The good news is I've confirmed that 
> launching tasks with Docker images *does* work by using Aurora linked against 
> 0.27.x but running Mesos 0.28.x in Vagrant.
> 1. In order to work around the setuid issues (i.e. task is launched as root, 
> but the executor cannot setuid because the role-user does not exist), I've 
> mounted /etc/passwd and /etc/group into the container and added a new flag, 
> `thermos_run_as_job_role`, to the scheduler. This flag is only used when 
> launching a task with a filesystem image, and causes us to add 
> `--execute-as-user ` to the thermos executor commandline.
> 1. The Mesos unified containerizer does not automatically create mount points 
> in the filesystem from the image. It expects the full path to the mount to 
> exist in the image. For /etc/passwd and /etc/groups this is not a problem, 
> but for the announcer acls file it was. I ended up moving the announcer acl 
> file into its own directory and mount that instead. In conjunction with this 
> I also had to modify our http_example Dockerfile to explicitly create that 
> mount point. A case could be made for sticking with the current path and just 
> creating an empty file in the image, I felt that creating an empty directory 
> was slightly less gross. This is tracked by 
> https://issues.apache.org/jira/browse/MESOS-5229.
> 1. The AppC image for end to end tests is created by running 
> [docker2aci](https://githu

Re: Review Request 46948: Adding mesos resource converter

2016-05-03 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46948/#review131602
---



Master (8a2fc4c) is red with this patch.
  ./build-support/jenkins/build.sh

Branch coverage is 0.7833495618305745, but must be greater than 0.835
:analyzeReport FAILED

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: 
> file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/tests/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
==

2: Task failed with an exception.
---
* What went wrong:
Execution failed for task ':analyzeReport'.
> Test coverage missing for org/apache/aurora/GuiceUtils$2
  Test coverage missing for 
org/apache/aurora/scheduler/stats/AsyncStatsModule$SlotSizeCounterService
  Test coverage missing for 
org/apache/aurora/scheduler/stats/AsyncStatsModule$TaskStatUpdaterService
  Test coverage missing for 
org/apache/aurora/scheduler/http/api/security/ShiroKerberosPermissiveAuthenticationFilter
  Test coverage missing for 
org/apache/aurora/scheduler/discovery/CuratorSingletonService
  Test coverage missing for 
org/apache/aurora/scheduler/discovery/CuratorSingletonService$1$1
  Test coverage missing for 
org/apache/aurora/scheduler/discovery/CuratorSingletonService$Advertiser
  Test coverage missing for 
org/apache/aurora/scheduler/discovery/CuratorSingletonService$1
  Test coverage missing for 
org/apache/aurora/scheduler/cron/quartz/CronLifecycle
  Test coverage missing for 
org/apache/aurora/scheduler/cron/quartz/AuroraCronJobFactory
  Test coverage missing for 
org/apache/aurora/scheduler/preemptor/PreemptorModule$PreemptorService
  Test coverage missing for 
org/apache/aurora/scheduler/storage/log/LogStorage$ScheduledExecutorSchedulingService
  Test coverage missing for 
org/apache/aurora/scheduler/SchedulerLifecycle$DefaultDelayedActions

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
==

BUILD FAILED

Total time: 13 mins 28.289 secs


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On May 3, 2016, 9:11 p.m., Maxim Khutornenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46948/
> ---
> 
> (Updated May 3, 2016, 9:11 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> A trivial patch to replace mesos type with the converter object in 
> `ResourceType`. 
> 
> This is mostly an attempt to offload this standalone change from the upcoming 
> larger diff that will introduce a unified approach of handling 
> `ResourceAggregate` and `ResourceSlot` data.
> 
> 
> Diffs
> -
> 
>   
> src/main/java/org/apache/aurora/scheduler/resources/MesosResourceConverter.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/resources/ResourceManager.java 
> 8b42bf0d2f4a4a19e52875440d825bc6ee553f3b 
>   src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java 
> ee2b51a587271457c5f1e0bcec6885bb4d0bc68f 
>   
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java
>  11395ecfdb701e7c6197b971482cb70a257fb06e 
>   src/main/java/org/apache/aurora/scheduler/resources/Resources.java 
> 36d1de85658afd5b0d933e833c7193c60afdfd9c 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java 
> dc7e97d1ea1f74594310bc346f18f2ed547f007d 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResourceAggregate.java
>  8c5d1f014cd448dbd2fdfa798ac8a3323855861f 
>   
> src/test/java/org/apache/aurora/scheduler/resources/MesosResourceConverterTest.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/resources/ResourceManagerTest.java 
> b6810b18677b9e030ad84a214ca7dc6746990d4f 
>   src/test/java/org/apache/aurora/scheduler/resources/ResourceTestUtil.java 
> 1583cefd568f4ed71f6953d5da0e2b2035bc 
>   
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java
>  78da84a9500de93025db5180c1d98a0669d62ec6 
>   src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java 
> dc9dc668e9f0a7484c62950834b5ae8cae2a1887 
> 
> Diff: https://reviews.apache.org/r/46948/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew -Pq build
> 
> 
> Thanks,
> 
> Maxim Khutornenko
> 
>

Re: Review Request 46948: Adding mesos resource converter

2016-05-03 Thread Maxim Khutornenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46948/#review131596
---



@ReviewBot retry

- Maxim Khutornenko


On May 3, 2016, 9:11 p.m., Maxim Khutornenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46948/
> ---
> 
> (Updated May 3, 2016, 9:11 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> A trivial patch to replace mesos type with the converter object in 
> `ResourceType`. 
> 
> This is mostly an attempt to offload this standalone change from the upcoming 
> larger diff that will introduce a unified approach of handling 
> `ResourceAggregate` and `ResourceSlot` data.
> 
> 
> Diffs
> -
> 
>   
> src/main/java/org/apache/aurora/scheduler/resources/MesosResourceConverter.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/resources/ResourceManager.java 
> 8b42bf0d2f4a4a19e52875440d825bc6ee553f3b 
>   src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java 
> ee2b51a587271457c5f1e0bcec6885bb4d0bc68f 
>   
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java
>  11395ecfdb701e7c6197b971482cb70a257fb06e 
>   src/main/java/org/apache/aurora/scheduler/resources/Resources.java 
> 36d1de85658afd5b0d933e833c7193c60afdfd9c 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java 
> dc7e97d1ea1f74594310bc346f18f2ed547f007d 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResourceAggregate.java
>  8c5d1f014cd448dbd2fdfa798ac8a3323855861f 
>   
> src/test/java/org/apache/aurora/scheduler/resources/MesosResourceConverterTest.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/resources/ResourceManagerTest.java 
> b6810b18677b9e030ad84a214ca7dc6746990d4f 
>   src/test/java/org/apache/aurora/scheduler/resources/ResourceTestUtil.java 
> 1583cefd568f4ed71f6953d5da0e2b2035bc 
>   
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java
>  78da84a9500de93025db5180c1d98a0669d62ec6 
>   src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java 
> dc9dc668e9f0a7484c62950834b5ae8cae2a1887 
> 
> Diff: https://reviews.apache.org/r/46948/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew -Pq build
> 
> 
> Thanks,
> 
> Maxim Khutornenko
> 
>

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

2016-05-03 Thread Maxim Khutornenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46835/#review131585
---




src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
 


Does this have to be bundled into this RB? We are still using this path in 
DEFAULT_CHECKPOINT_ROOT on the thermos side. Perhaps, have a more thorough 
cleanup in a separate patch?



src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java (line 201)


This name is no longer accurate given that we actually use 
ContainerInfo.Builder inside.



src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java 
(line 83)


Shouldn't this and `DBImage` go into `DBContainer` now that `TaskConfig` 
does not have an `image` property?



src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
(line 422)


Wouldn't a fake imageId suffice here?


- Maxim Khutornenko


On May 3, 2016, 10:04 p.m., Joshua Cohen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46835/
> ---
> 
> (Updated May 3, 2016, 10:04 p.m.)
> 
> 
> Review request for Aurora, John Sirois, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1636, AURORA-1637, AURORA-1638, and AURORA-1639
> https://issues.apache.org/jira/browse/AURORA-1636
> https://issues.apache.org/jira/browse/AURORA-1637
> https://issues.apache.org/jira/browse/AURORA-1638
> https://issues.apache.org/jira/browse/AURORA-1639
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> A few notes:
> 
> 1. It's not possible to configure Mesos 0.27.x to launch docker tasks due to 
> a bug in parsing the docker_store_dir flag. Fixed here: 
> https://reviews.apache.org/r/43451/ but has not been backported to Mesos 
> 0.27. This means we can only launch tasks that use AppC images until we 
> upgrade our Mesos dependency to 0.28.x. The good news is I've confirmed that 
> launching tasks with Docker images *does* work by using Aurora linked against 
> 0.27.x but running Mesos 0.28.x in Vagrant.
> 1. In order to work around the setuid issues (i.e. task is launched as root, 
> but the executor cannot setuid because the role-user does not exist), I've 
> mounted /etc/passwd and /etc/group into the container and added a new flag, 
> `thermos_run_as_job_role`, to the scheduler. This flag is only used when 
> launching a task with a filesystem image, and causes us to add 
> `--execute-as-user ` to the thermos executor commandline.
> 1. The Mesos unified containerizer does not automatically create mount points 
> in the filesystem from the image. It expects the full path to the mount to 
> exist in the image. For /etc/passwd and /etc/groups this is not a problem, 
> but for the announcer acls file it was. I ended up moving the announcer acl 
> file into its own directory and mount that instead. In conjunction with this 
> I also had to modify our http_example Dockerfile to explicitly create that 
> mount point. A case could be made for sticking with the current path and just 
> creating an empty file in the image, I felt that creating an empty directory 
> was slightly less gross. This is tracked by 
> https://issues.apache.org/jira/browse/MESOS-5229.
> 1. The AppC image for end to end tests is created by running 
> [docker2aci](https://github.com/appc/docker2aci) on our http_example docker 
> image. The base box needed to be upgraded to add this utility. I haven't 
> published the new base box yet even though I've updated the Vagrantfile to 
> point to version 6. Once this review has been approved and I'm sure there's 
> no further changes that need to be made I'll publish the base box before 
> committing.
> 
> 
> Diffs
> -
> 
>   3rdparty/python/requirements.txt 666c4ae487332f01380cfce76f0d97e2c6049c8e 
>   RELEASE-NOTES.md 7a37d0d69f688bece624628fe5b98efc85d506a2 
>   Vagrantfile 3f126ee348d0f95d6f159b62280de79f41e87e2e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 38470951e4482753fcada109ab12546a2fb146ce 
>   build-support/packer/build.sh 76197c31c365aa3d8a67049da40b2976c1e25d22 
>   docs/reference/configuration.md 9fcfdfcd9ab793e888ca2bba2035d5122142a5ab 
>   docs/reference/scheduler-configuration.md 
> d2262f79edfde23eccd87bae7f1cf319b63b1103 
>   examples/vagrant/announcer-auth.json  
>   examples/vagrant/mesos_config/etc_mesos-slave/appc_store_dir PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/image_providers PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/image_provi

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

2016-05-03 Thread Stephan Erb


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46835/#review131577
---




src/main/python/apache/aurora/config/schema/base.py (line 142)


This can be removed, I guess.


- Stephan Erb


On May 4, 2016, 12:04 a.m., Joshua Cohen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46835/
> ---
> 
> (Updated May 4, 2016, 12:04 a.m.)
> 
> 
> Review request for Aurora, John Sirois, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1636, AURORA-1637, AURORA-1638, and AURORA-1639
> https://issues.apache.org/jira/browse/AURORA-1636
> https://issues.apache.org/jira/browse/AURORA-1637
> https://issues.apache.org/jira/browse/AURORA-1638
> https://issues.apache.org/jira/browse/AURORA-1639
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> A few notes:
> 
> 1. It's not possible to configure Mesos 0.27.x to launch docker tasks due to 
> a bug in parsing the docker_store_dir flag. Fixed here: 
> https://reviews.apache.org/r/43451/ but has not been backported to Mesos 
> 0.27. This means we can only launch tasks that use AppC images until we 
> upgrade our Mesos dependency to 0.28.x. The good news is I've confirmed that 
> launching tasks with Docker images *does* work by using Aurora linked against 
> 0.27.x but running Mesos 0.28.x in Vagrant.
> 1. In order to work around the setuid issues (i.e. task is launched as root, 
> but the executor cannot setuid because the role-user does not exist), I've 
> mounted /etc/passwd and /etc/group into the container and added a new flag, 
> `thermos_run_as_job_role`, to the scheduler. This flag is only used when 
> launching a task with a filesystem image, and causes us to add 
> `--execute-as-user ` to the thermos executor commandline.
> 1. The Mesos unified containerizer does not automatically create mount points 
> in the filesystem from the image. It expects the full path to the mount to 
> exist in the image. For /etc/passwd and /etc/groups this is not a problem, 
> but for the announcer acls file it was. I ended up moving the announcer acl 
> file into its own directory and mount that instead. In conjunction with this 
> I also had to modify our http_example Dockerfile to explicitly create that 
> mount point. A case could be made for sticking with the current path and just 
> creating an empty file in the image, I felt that creating an empty directory 
> was slightly less gross. This is tracked by 
> https://issues.apache.org/jira/browse/MESOS-5229.
> 1. The AppC image for end to end tests is created by running 
> [docker2aci](https://github.com/appc/docker2aci) on our http_example docker 
> image. The base box needed to be upgraded to add this utility. I haven't 
> published the new base box yet even though I've updated the Vagrantfile to 
> point to version 6. Once this review has been approved and I'm sure there's 
> no further changes that need to be made I'll publish the base box before 
> committing.
> 
> 
> Diffs
> -
> 
>   3rdparty/python/requirements.txt 666c4ae487332f01380cfce76f0d97e2c6049c8e 
>   RELEASE-NOTES.md 7a37d0d69f688bece624628fe5b98efc85d506a2 
>   Vagrantfile 3f126ee348d0f95d6f159b62280de79f41e87e2e 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> 38470951e4482753fcada109ab12546a2fb146ce 
>   build-support/packer/build.sh 76197c31c365aa3d8a67049da40b2976c1e25d22 
>   docs/reference/configuration.md 9fcfdfcd9ab793e888ca2bba2035d5122142a5ab 
>   docs/reference/scheduler-configuration.md 
> d2262f79edfde23eccd87bae7f1cf319b63b1103 
>   examples/vagrant/announcer-auth.json  
>   examples/vagrant/mesos_config/etc_mesos-slave/appc_store_dir PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/image_providers PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/image_provisioner_backend 
> PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/isolation PRE-CREATION 
>   examples/vagrant/upstart/aurora-scheduler.conf 
> 084016abc169ed82b7ed00f5d14aea2e0ff38a49 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
>  9a15a4b5c1cdc3efbf0588e8bd81636b66e938ef 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
>  32f2fa90b21189180e2bcd65a3cebf13f6551646 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> b325106c7f45b1ad1657221aaa39e3a428719ab0 
>   src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java 
> 9aadcebf547bd1eb4b4e238507e27ae2b699f473 
>   src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 
> 9eadf70240738030593a819a4a248588c9d1c35b 
>   
> src/main/j

Re: Review Request 46596: Command line flag to change FrameworkInfo.name

2016-05-03 Thread Stephan Erb


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46596/
---

(Updated May 4, 2016, 12:24 a.m.)


Review request for Aurora, Maxim Khutornenko and Zameer Manji.


Changes
---

As discussed in IRC a couple of days ago, I am opting for the additional 
deprecation cycle.


Bugs: AURORA-945
https://issues.apache.org/jira/browse/AURORA-945


Repository: aurora


Description (updated)
---

This commit introduces a new command line flag `-framework_name` and changes 
the value from the hardcoded 'TwitterScheduler' to 'aurora'.

This change is non-intrusive as 
https://issues.apache.org/jira/browse/MESOS-2614 has been fixed. The new value 
will be shown on the UI and used for Mesos DNS.


Diffs (updated)
-

  RELEASE-NOTES.md 7a37d0d69f688bece624628fe5b98efc85d506a2 
  docs/features/service-discovery.md f242730fc16103903167182372ff6903127d73f5 
  docs/reference/scheduler-configuration.md 
d2262f79edfde23eccd87bae7f1cf319b63b1103 
  
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
 5c6cdd2b4c7ee92eb7d8e1649a95cad3035efec0 
  
src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
 dc964b8dac1b8e21e86ad331216c57fa11abeb6d 

Diff: https://reviews.apache.org/r/46596/diff/


Testing
---

* ./gradlew -Pq build
* Changed the framework name in vagrant. Aurora happily re-registered without a 
fallout.


Thanks,

Stephan Erb

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

2016-05-03 Thread Aurora ReviewBot

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46835/#review131573
---

Master (d8d1c8d) is red with this patch.
  ./build-support/jenkins/build.sh

:generateBuildProperties
:processResources
:classes
:jar
:startScripts
:distTar
:distZip
:assemble
:compileJmhJavaNote: 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java
 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

:processJmhResources UP-TO-DATE
:jmhClasses
:checkstyleJmh
:jsHint
:checkstyleMain
:compileTestJava/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/thrift/aop/MockDecoratedThrift.java:38:
 Note: Wrote forwarder 
org.apache.aurora.scheduler.thrift.aop.MockDecoratedThriftForwarder
@Forward(AnnotatedAuroraAdmin.class)
^
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

:processTestResources
:testClasses
:checkstyleTest[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java:54:8:
 error: Unused import - org.apache.aurora.gen.TaskConfig.
[ant:checkstyle] 
/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java:63:8:
 error: Unused import - 
org.apache.aurora.scheduler.storage.entities.ITaskConfig.
 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':checkstyleTest'.
> Checkstyle rule violations were found. See the report at: 
> file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/checkstyle/test.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 4 mins 6.199 secs

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot

On May 3, 2016, 10:04 p.m., Joshua Cohen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46835/
> ---
> 
> (Updated May 3, 2016, 10:04 p.m.)
> 
> 
> Review request for Aurora, John Sirois, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1636, AURORA-1637, AURORA-1638, and AURORA-1639
> https://issues.apache.org/jira/browse/AURORA-1636
> https://issues.apache.org/jira/browse/AURORA-1637
> https://issues.apache.org/jira/browse/AURORA-1638
> https://issues.apache.org/jira/browse/AURORA-1639
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> A few notes:
> 
> 1. It's not possible to configure Mesos 0.27.x to launch docker tasks due to 
> a bug in parsing the docker_store_dir flag. Fixed here: 
> https://reviews.apache.org/r/43451/ but has not been backported to Mesos 
> 0.27. This means we can only launch tasks that use AppC images until we 
> upgrade our Mesos dependency to 0.28.x. The good news is I've confirmed that 
> launching tasks with Docker images *does* work by using Aurora linked against 
> 0.27.x but running Mesos 0.28.x in Vagrant.
> 1. In order to work around the setuid issues (i.e. task is launched as root, 
> but the executor cannot setuid because the role-user does not exist), I've 
> mounted /etc/passwd and /etc/group into the container and added a new flag, 
> `thermos_run_as_job_role`, to the scheduler. This flag is only used when 
> launching a task with a filesystem image, and causes us to add 
> `--execute-as-user ` to the thermos executor commandline.
> 1. The Mesos unified containerizer does not automatically create mount points 
> in the filesystem from the image. It expects the full path to the mount to 
> exist in the image. For /etc/passwd and /etc/groups this is not a problem, 
> but for the announcer acls file it was. I ended up moving the announcer acl 
> file into its own directory and mount that instead. In conjunction with this 
> I also had to modify our http_example Dockerfile to explicitly create that 
> mount point. A case could be made for sticking with the current path and just 
> creating an empty file in the image, I felt that creating an empty directory 
> was slightly less gross. This is tracked by 
> https://issues.apache.org/jira/browse/MESOS-5229.
> 1. The AppC image for end to end tests is created by running 
> [docker2aci](https://github.com/appc/docker2aci) on our http_example docker 
> image. The base box needed to be upgraded to add this utility. I haven't 
> published the new base box yet even though I've updated the Vagrantfile to 
> point to version 6. Once this review has been approved and I'm sure there's 
> no further changes that need to be ma

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

2016-05-03 Thread Joshua Cohen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46835/
---

(Updated May 3, 2016, 10:04 p.m.)


Review request for Aurora, John Sirois, Maxim Khutornenko, and Bill Farner.


Changes
---

- Upgrade to pystachio 0.8.1 to pick up the new Choice type.
- Rework client configuration and thrift to make image a property of the Mesos 
container (made feasible by the pystachio Choice type).
- Remove the `thermos_run_as_job_role` arg.
- Remove the no longer necessary `thermos_observer_root` command line arg from 
the scheduler.


Bugs: AURORA-1636, AURORA-1637, AURORA-1638, and AURORA-1639
https://issues.apache.org/jira/browse/AURORA-1636
https://issues.apache.org/jira/browse/AURORA-1637
https://issues.apache.org/jira/browse/AURORA-1638
https://issues.apache.org/jira/browse/AURORA-1639


Repository: aurora


Description
---

A few notes:

1. It's not possible to configure Mesos 0.27.x to launch docker tasks due to a 
bug in parsing the docker_store_dir flag. Fixed here: 
https://reviews.apache.org/r/43451/ but has not been backported to Mesos 0.27. 
This means we can only launch tasks that use AppC images until we upgrade our 
Mesos dependency to 0.28.x. The good news is I've confirmed that launching 
tasks with Docker images *does* work by using Aurora linked against 0.27.x but 
running Mesos 0.28.x in Vagrant.
1. In order to work around the setuid issues (i.e. task is launched as root, 
but the executor cannot setuid because the role-user does not exist), I've 
mounted /etc/passwd and /etc/group into the container and added a new flag, 
`thermos_run_as_job_role`, to the scheduler. This flag is only used when 
launching a task with a filesystem image, and causes us to add 
`--execute-as-user ` to the thermos executor commandline.
1. The Mesos unified containerizer does not automatically create mount points 
in the filesystem from the image. It expects the full path to the mount to 
exist in the image. For /etc/passwd and /etc/groups this is not a problem, but 
for the announcer acls file it was. I ended up moving the announcer acl file 
into its own directory and mount that instead. In conjunction with this I also 
had to modify our http_example Dockerfile to explicitly create that mount 
point. A case could be made for sticking with the current path and just 
creating an empty file in the image, I felt that creating an empty directory 
was slightly less gross. This is tracked by 
https://issues.apache.org/jira/browse/MESOS-5229.
1. The AppC image for end to end tests is created by running 
[docker2aci](https://github.com/appc/docker2aci) on our http_example docker 
image. The base box needed to be upgraded to add this utility. I haven't 
published the new base box yet even though I've updated the Vagrantfile to 
point to version 6. Once this review has been approved and I'm sure there's no 
further changes that need to be made I'll publish the base box before 
committing.


Diffs (updated)
-

  3rdparty/python/requirements.txt 666c4ae487332f01380cfce76f0d97e2c6049c8e 
  RELEASE-NOTES.md 7a37d0d69f688bece624628fe5b98efc85d506a2 
  Vagrantfile 3f126ee348d0f95d6f159b62280de79f41e87e2e 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
38470951e4482753fcada109ab12546a2fb146ce 
  build-support/packer/build.sh 76197c31c365aa3d8a67049da40b2976c1e25d22 
  docs/reference/configuration.md 9fcfdfcd9ab793e888ca2bba2035d5122142a5ab 
  docs/reference/scheduler-configuration.md 
d2262f79edfde23eccd87bae7f1cf319b63b1103 
  examples/vagrant/announcer-auth.json  
  examples/vagrant/mesos_config/etc_mesos-slave/appc_store_dir PRE-CREATION 
  examples/vagrant/mesos_config/etc_mesos-slave/image_providers PRE-CREATION 
  examples/vagrant/mesos_config/etc_mesos-slave/image_provisioner_backend 
PRE-CREATION 
  examples/vagrant/mesos_config/etc_mesos-slave/isolation PRE-CREATION 
  examples/vagrant/upstart/aurora-scheduler.conf 
084016abc169ed82b7ed00f5d14aea2e0ff38a49 
  
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
 9a15a4b5c1cdc3efbf0588e8bd81636b66e938ef 
  
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
 32f2fa90b21189180e2bcd65a3cebf13f6551646 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
b325106c7f45b1ad1657221aaa39e3a428719ab0 
  src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java 
9aadcebf547bd1eb4b4e238507e27ae2b699f473 
  src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 
9eadf70240738030593a819a4a248588c9d1c35b 
  src/main/java/org/apache/aurora/scheduler/storage/db/views/DbTaskConfig.java 
a7523c4b258030bcfb2e457b083242ffa865a98a 
  src/main/python/apache/aurora/config/schema/base.py 
00be8747d70dbf1cb370f09536588f8602d8fcce 
  src/main/python/apache/aurora/config/thrift.py 
928ca9313b2c206

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

2016-05-03 Thread Joshua Cohen



> On May 3, 2016, 8:39 p.m., Stephan Erb wrote:
> > RELEASE-NOTES.md, line 23
> > 
> >
> > How would I do the mounting? Via the global container mounts scheduler 
> > option?
> > 
> > So, once we have mounted the passwd and group file, is 
> > `thermos_run_as_job_role` still needed? Shouldn't that allow us to use the 
> > normal setuid of thermos?

> How would I do the mounting? Via the global container mounts scheduler option?

Yes (can see our aurora-scheduler.conf for the vagrant image for an example).

> So, once we have mounted the passwd and group file, is 
> thermos_run_as_job_role still needed? Shouldn't that allow us to use the 
> normal setuid of thermos?

You're right, it's actually not needed. The root cause of the problem that made 
me think this was required was that the `DockerDirectorySandbox` (now the 
`FileSystemImageSandbox`) did not take the user as a parameter, and instead 
always passed `None`. As part of adding support for `thermos_run_as_job_role` I 
added that arg to the constructor and didn't rethink the need for the scheduler 
arg entirely. Thanks for catching this!


- Joshua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46835/#review131547
---


On April 29, 2016, 4:22 p.m., Joshua Cohen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46835/
> ---
> 
> (Updated April 29, 2016, 4:22 p.m.)
> 
> 
> Review request for Aurora, John Sirois, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1636, AURORA-1637, AURORA-1638, and AURORA-1639
> https://issues.apache.org/jira/browse/AURORA-1636
> https://issues.apache.org/jira/browse/AURORA-1637
> https://issues.apache.org/jira/browse/AURORA-1638
> https://issues.apache.org/jira/browse/AURORA-1639
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> A few notes:
> 
> 1. It's not possible to configure Mesos 0.27.x to launch docker tasks due to 
> a bug in parsing the docker_store_dir flag. Fixed here: 
> https://reviews.apache.org/r/43451/ but has not been backported to Mesos 
> 0.27. This means we can only launch tasks that use AppC images until we 
> upgrade our Mesos dependency to 0.28.x. The good news is I've confirmed that 
> launching tasks with Docker images *does* work by using Aurora linked against 
> 0.27.x but running Mesos 0.28.x in Vagrant.
> 1. In order to work around the setuid issues (i.e. task is launched as root, 
> but the executor cannot setuid because the role-user does not exist), I've 
> mounted /etc/passwd and /etc/group into the container and added a new flag, 
> `thermos_run_as_job_role`, to the scheduler. This flag is only used when 
> launching a task with a filesystem image, and causes us to add 
> `--execute-as-user ` to the thermos executor commandline.
> 1. The Mesos unified containerizer does not automatically create mount points 
> in the filesystem from the image. It expects the full path to the mount to 
> exist in the image. For /etc/passwd and /etc/groups this is not a problem, 
> but for the announcer acls file it was. I ended up moving the announcer acl 
> file into its own directory and mount that instead. In conjunction with this 
> I also had to modify our http_example Dockerfile to explicitly create that 
> mount point. A case could be made for sticking with the current path and just 
> creating an empty file in the image, I felt that creating an empty directory 
> was slightly less gross. This is tracked by 
> https://issues.apache.org/jira/browse/MESOS-5229.
> 1. The AppC image for end to end tests is created by running 
> [docker2aci](https://github.com/appc/docker2aci) on our http_example docker 
> image. The base box needed to be upgraded to add this utility. I haven't 
> published the new base box yet even though I've updated the Vagrantfile to 
> point to version 6. Once this review has been approved and I'm sure there's 
> no further changes that need to be made I'll publish the base box before 
> committing.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 7a37d0d69f688bece624628fe5b98efc85d506a2 
>   Vagrantfile 3f126ee348d0f95d6f159b62280de79f41e87e2e 
>   build-support/packer/build.sh 76197c31c365aa3d8a67049da40b2976c1e25d22 
>   docs/reference/configuration.md 9fcfdfcd9ab793e888ca2bba2035d5122142a5ab 
>   docs/reference/scheduler-configuration.md 
> d2262f79edfde23eccd87bae7f1cf319b63b1103 
>   examples/vagrant/announcer-auth.json  
>   examples/vagrant/mesos_config/etc_mesos-slave/appc_store_dir PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/image_providers PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/imag

Re: Review Request 46948: Adding mesos resource converter

2016-05-03 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46948/#review131561
---



Master (d8d1c8d) is red with this patch.
  ./build-support/jenkins/build.sh

+ date
Tue May  3 21:21:26 UTC 2016
+ ./gradlew -Pq clean build
./gradlew: fork: retry: No child processes
./gradlew: fork: retry: No child processes
./gradlew: fork: retry: No child processes
./gradlew: fork: retry: No child processes
./gradlew: fork: Resource temporarily unavailable


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On May 3, 2016, 9:11 p.m., Maxim Khutornenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46948/
> ---
> 
> (Updated May 3, 2016, 9:11 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> A trivial patch to replace mesos type with the converter object in 
> `ResourceType`. 
> 
> This is mostly an attempt to offload this standalone change from the upcoming 
> larger diff that will introduce a unified approach of handling 
> `ResourceAggregate` and `ResourceSlot` data.
> 
> 
> Diffs
> -
> 
>   
> src/main/java/org/apache/aurora/scheduler/resources/MesosResourceConverter.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/resources/ResourceManager.java 
> 8b42bf0d2f4a4a19e52875440d825bc6ee553f3b 
>   src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java 
> ee2b51a587271457c5f1e0bcec6885bb4d0bc68f 
>   
> src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java
>  11395ecfdb701e7c6197b971482cb70a257fb06e 
>   src/main/java/org/apache/aurora/scheduler/resources/Resources.java 
> 36d1de85658afd5b0d933e833c7193c60afdfd9c 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java 
> dc7e97d1ea1f74594310bc346f18f2ed547f007d 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResourceAggregate.java
>  8c5d1f014cd448dbd2fdfa798ac8a3323855861f 
>   
> src/test/java/org/apache/aurora/scheduler/resources/MesosResourceConverterTest.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/resources/ResourceManagerTest.java 
> b6810b18677b9e030ad84a214ca7dc6746990d4f 
>   src/test/java/org/apache/aurora/scheduler/resources/ResourceTestUtil.java 
> 1583cefd568f4ed71f6953d5da0e2b2035bc 
>   
> src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java
>  78da84a9500de93025db5180c1d98a0669d62ec6 
>   src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java 
> dc9dc668e9f0a7484c62950834b5ae8cae2a1887 
> 
> Diff: https://reviews.apache.org/r/46948/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew -Pq build
> 
> 
> Thanks,
> 
> Maxim Khutornenko
> 
>

Review Request 46948: Adding mesos resource converter

2016-05-03 Thread Maxim Khutornenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46948/
---

Review request for Aurora, Joshua Cohen and Zameer Manji.


Repository: aurora


Description
---

A trivial patch to replace mesos type with the converter object in 
`ResourceType`. 

This is mostly an attempt to offload this standalone change from the upcoming 
larger diff that will introduce a unified approach of handling 
`ResourceAggregate` and `ResourceSlot` data.


Diffs
-

  
src/main/java/org/apache/aurora/scheduler/resources/MesosResourceConverter.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/resources/ResourceManager.java 
8b42bf0d2f4a4a19e52875440d825bc6ee553f3b 
  src/main/java/org/apache/aurora/scheduler/resources/ResourceType.java 
ee2b51a587271457c5f1e0bcec6885bb4d0bc68f 
  
src/main/java/org/apache/aurora/scheduler/resources/ResourceTypeConverter.java 
11395ecfdb701e7c6197b971482cb70a257fb06e 
  src/main/java/org/apache/aurora/scheduler/resources/Resources.java 
36d1de85658afd5b0d933e833c7193c60afdfd9c 
  src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResource.java 
dc7e97d1ea1f74594310bc346f18f2ed547f007d 
  
src/main/java/org/apache/aurora/scheduler/storage/db/views/DBResourceAggregate.java
 8c5d1f014cd448dbd2fdfa798ac8a3323855861f 
  
src/test/java/org/apache/aurora/scheduler/resources/MesosResourceConverterTest.java
 PRE-CREATION 
  src/test/java/org/apache/aurora/scheduler/resources/ResourceManagerTest.java 
b6810b18677b9e030ad84a214ca7dc6746990d4f 
  src/test/java/org/apache/aurora/scheduler/resources/ResourceTestUtil.java 
1583cefd568f4ed71f6953d5da0e2b2035bc 
  
src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeConverterTest.java
 78da84a9500de93025db5180c1d98a0669d62ec6 
  src/test/java/org/apache/aurora/scheduler/resources/ResourceTypeTest.java 
dc9dc668e9f0a7484c62950834b5ae8cae2a1887 

Diff: https://reviews.apache.org/r/46948/diff/


Testing
---

./gradlew -Pq build


Thanks,

Maxim Khutornenko

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

2016-05-03 Thread Stephan Erb


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46835/#review131547
---




RELEASE-NOTES.md (line 23)


How would I do the mounting? Via the global container mounts scheduler 
option?

So, once we have mounted the passwd and group file, is 
`thermos_run_as_job_role` still needed? Shouldn't that allow us to use the 
normal setuid of thermos?


- Stephan Erb


On April 29, 2016, 6:22 p.m., Joshua Cohen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46835/
> ---
> 
> (Updated April 29, 2016, 6:22 p.m.)
> 
> 
> Review request for Aurora, John Sirois, Maxim Khutornenko, and Bill Farner.
> 
> 
> Bugs: AURORA-1636, AURORA-1637, AURORA-1638, and AURORA-1639
> https://issues.apache.org/jira/browse/AURORA-1636
> https://issues.apache.org/jira/browse/AURORA-1637
> https://issues.apache.org/jira/browse/AURORA-1638
> https://issues.apache.org/jira/browse/AURORA-1639
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> A few notes:
> 
> 1. It's not possible to configure Mesos 0.27.x to launch docker tasks due to 
> a bug in parsing the docker_store_dir flag. Fixed here: 
> https://reviews.apache.org/r/43451/ but has not been backported to Mesos 
> 0.27. This means we can only launch tasks that use AppC images until we 
> upgrade our Mesos dependency to 0.28.x. The good news is I've confirmed that 
> launching tasks with Docker images *does* work by using Aurora linked against 
> 0.27.x but running Mesos 0.28.x in Vagrant.
> 1. In order to work around the setuid issues (i.e. task is launched as root, 
> but the executor cannot setuid because the role-user does not exist), I've 
> mounted /etc/passwd and /etc/group into the container and added a new flag, 
> `thermos_run_as_job_role`, to the scheduler. This flag is only used when 
> launching a task with a filesystem image, and causes us to add 
> `--execute-as-user ` to the thermos executor commandline.
> 1. The Mesos unified containerizer does not automatically create mount points 
> in the filesystem from the image. It expects the full path to the mount to 
> exist in the image. For /etc/passwd and /etc/groups this is not a problem, 
> but for the announcer acls file it was. I ended up moving the announcer acl 
> file into its own directory and mount that instead. In conjunction with this 
> I also had to modify our http_example Dockerfile to explicitly create that 
> mount point. A case could be made for sticking with the current path and just 
> creating an empty file in the image, I felt that creating an empty directory 
> was slightly less gross. This is tracked by 
> https://issues.apache.org/jira/browse/MESOS-5229.
> 1. The AppC image for end to end tests is created by running 
> [docker2aci](https://github.com/appc/docker2aci) on our http_example docker 
> image. The base box needed to be upgraded to add this utility. I haven't 
> published the new base box yet even though I've updated the Vagrantfile to 
> point to version 6. Once this review has been approved and I'm sure there's 
> no further changes that need to be made I'll publish the base box before 
> committing.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 7a37d0d69f688bece624628fe5b98efc85d506a2 
>   Vagrantfile 3f126ee348d0f95d6f159b62280de79f41e87e2e 
>   build-support/packer/build.sh 76197c31c365aa3d8a67049da40b2976c1e25d22 
>   docs/reference/configuration.md 9fcfdfcd9ab793e888ca2bba2035d5122142a5ab 
>   docs/reference/scheduler-configuration.md 
> d2262f79edfde23eccd87bae7f1cf319b63b1103 
>   examples/vagrant/announcer-auth.json  
>   examples/vagrant/mesos_config/etc_mesos-slave/appc_store_dir PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/image_providers PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/image_provisioner_backend 
> PRE-CREATION 
>   examples/vagrant/mesos_config/etc_mesos-slave/isolation PRE-CREATION 
>   examples/vagrant/upstart/aurora-scheduler.conf 
> 084016abc169ed82b7ed00f5d14aea2e0ff38a49 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
>  32f2fa90b21189180e2bcd65a3cebf13f6551646 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
>  501e6431f21822d9816952377546586da02ce42a 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> b325106c7f45b1ad1657221aaa39e3a428719ab0 
>   src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java 
> 9aadcebf547bd1eb4b4e238507e27ae2b699f473 
>   src/main/python/apache/aurora/config/schema/base.py 
> 00be8747d70dbf1cb370f09536588f8602d8fcce 
>   src/main/python/apache/auror

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

Re: Review Request 46948: Adding mesos resource converter

Re: Review Request 46948: Adding mesos resource converter

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

Re: Review Request 46596: Command line flag to change FrameworkInfo.name

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

Re: Review Request 46948: Adding mesos resource converter

Review Request 46948: Adding mesos resource converter

Re: Review Request 46835: Add client and scheduler support for launching tasks using the Mesos unified containerizer

15 matches

Site Navigation

Mail list logo

Footer information