Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 14, 2015, 7:55 p.m., Bill Farner wrote: All nits this round. Last issue for me, as you point out, is validation/restriction of `ContainerType` in `ConfigurationManager. Doing a pass on test coverage in the meantime. Bill Farner wrote: Test coverage LGTM when supplemented with https://reviews.apache.org/r/29827/ Steve Niemitz wrote: Is there any need to validate anymore now that the volume configuration is gone? That's what was going to be validated correct? I would like to see an argument to specify which `ContainerType`s the scheduler should allow. This way the scheduler will not fatefully accept jobs and launch tasks that are not supported by the slave's command line and environment. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68098 --- On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 8:20 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py
Re: Review Request 29873: Fixing batched kill task filtering.
On Jan. 14, 2015, 7:26 p.m., Bill Farner wrote: Also, dropped active instance validation from both updater calls as it was hiding a legitimate feature: add new job instances with instance_spec. I may have missed it - do unit tests verify this new behavior? The updater.py tests do. Since the functionality of validating instance spec is effectively removed, I don't see much benefit adding tests to validate the absent feature. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29873/#review68082 --- On Jan. 14, 2015, 1:16 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29873/ --- (Updated Jan. 14, 2015, 1:16 a.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-996 https://issues.apache.org/jira/browse/AURORA-996 Repository: aurora Description --- Fixed job status quering in client context to return active tasks where needed. Also, dropped active instance validation from both updater calls as it was hiding a legitimate feature: add new job instances with instance_spec. Diffs - src/main/python/apache/aurora/client/cli/context.py 93587c616afb0b7493a509197361cd76af2e2c97 src/main/python/apache/aurora/client/cli/jobs.py 508c9be556998e47bddcec8dee43f1595497b354 src/main/python/apache/aurora/client/cli/update.py a1617325f08ca252bdba38618aef141504ba7272 src/test/python/apache/aurora/client/cli/test_kill.py b475d737ede8ff0a669a9a9229196a76b43b46b6 src/test/python/apache/aurora/client/cli/test_restart.py a532ead256869c620e6bd96886ce9681b3423d0c Diff: https://reviews.apache.org/r/29873/diff/ Testing --- ./pants src/test/python:all vagrant@vagrant-ubuntu-trusty-64:~$ aurora job killall devcluster/www-data/prod/hello INFO] Checking status of devcluster/www-data/prod/hello INFO] No tasks to kill found for job devcluster/www-data/prod/hello Job killall succeeded Thanks, Maxim Khutornenko
Re: Review Request 29873: Fixing batched kill task filtering.
On Jan. 14, 2015, 7:27 p.m., Bill Farner wrote: src/test/python/apache/aurora/client/cli/test_kill.py, line 146 https://reviews.apache.org/r/29873/diff/1/?file=820323#file820323line146 While you're in here, can you take a stab at renaming `create_mock_task` to `create_task`, and not using mocks (ditto for `create_scheduled_tasks`)? This aligns with the desired pattern of avoiding unnecessary use of mocks. You can borrow from `test_inspect.py` for an example: https://reviews.apache.org/r/29696 Sure, will do. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29873/#review68083 --- On Jan. 14, 2015, 1:16 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29873/ --- (Updated Jan. 14, 2015, 1:16 a.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-996 https://issues.apache.org/jira/browse/AURORA-996 Repository: aurora Description --- Fixed job status quering in client context to return active tasks where needed. Also, dropped active instance validation from both updater calls as it was hiding a legitimate feature: add new job instances with instance_spec. Diffs - src/main/python/apache/aurora/client/cli/context.py 93587c616afb0b7493a509197361cd76af2e2c97 src/main/python/apache/aurora/client/cli/jobs.py 508c9be556998e47bddcec8dee43f1595497b354 src/main/python/apache/aurora/client/cli/update.py a1617325f08ca252bdba38618aef141504ba7272 src/test/python/apache/aurora/client/cli/test_kill.py b475d737ede8ff0a669a9a9229196a76b43b46b6 src/test/python/apache/aurora/client/cli/test_restart.py a532ead256869c620e6bd96886ce9681b3423d0c Diff: https://reviews.apache.org/r/29873/diff/ Testing --- ./pants src/test/python:all vagrant@vagrant-ubuntu-trusty-64:~$ aurora job killall devcluster/www-data/prod/hello INFO] Checking status of devcluster/www-data/prod/hello INFO] No tasks to kill found for job devcluster/www-data/prod/hello Job killall succeeded Thanks, Maxim Khutornenko
Re: Review Request 29873: Fixing batched kill task filtering.
On Jan. 14, 2015, 7:21 p.m., Bill Farner wrote: src/main/python/apache/aurora/client/cli/context.py, line 214 https://reviews.apache.org/r/29873/diff/1/?file=820320#file820320line214 Seems like it would make sense for this to be the default behavior of `get_job_status`. I can't imagine why the default should be to fetch the active tasks plus some arbitrary history (since history is pruned) of dead instances. Meta: we should ~always do filtering like this server-side if possible. In this case, you should pass ACTIVE_STATES as a query parameter. `get_status_for_jobs` is also guilty of this, but would benefit from the proposd default behavior of `get_job_status` as well. The get_job_status is used by aurora job status to pull all available instances (including terminal ones). I was split between using a pre- and post- query filters and decided for the latter as a simpler one. The perf gain is quite neglible given the limited history we persist but I am happy to reconsider if you think it's worth it. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29873/#review68078 --- On Jan. 14, 2015, 1:16 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29873/ --- (Updated Jan. 14, 2015, 1:16 a.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-996 https://issues.apache.org/jira/browse/AURORA-996 Repository: aurora Description --- Fixed job status quering in client context to return active tasks where needed. Also, dropped active instance validation from both updater calls as it was hiding a legitimate feature: add new job instances with instance_spec. Diffs - src/main/python/apache/aurora/client/cli/context.py 93587c616afb0b7493a509197361cd76af2e2c97 src/main/python/apache/aurora/client/cli/jobs.py 508c9be556998e47bddcec8dee43f1595497b354 src/main/python/apache/aurora/client/cli/update.py a1617325f08ca252bdba38618aef141504ba7272 src/test/python/apache/aurora/client/cli/test_kill.py b475d737ede8ff0a669a9a9229196a76b43b46b6 src/test/python/apache/aurora/client/cli/test_restart.py a532ead256869c620e6bd96886ce9681b3423d0c Diff: https://reviews.apache.org/r/29873/diff/ Testing --- ./pants src/test/python:all vagrant@vagrant-ubuntu-trusty-64:~$ aurora job killall devcluster/www-data/prod/hello INFO] Checking status of devcluster/www-data/prod/hello INFO] No tasks to kill found for job devcluster/www-data/prod/hello Job killall succeeded Thanks, Maxim Khutornenko
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 14, 2015, 7:55 p.m., Bill Farner wrote: All nits this round. Last issue for me, as you point out, is validation/restriction of `ContainerType` in `ConfigurationManager. Doing a pass on test coverage in the meantime. Bill Farner wrote: Test coverage LGTM when supplemented with https://reviews.apache.org/r/29827/ Steve Niemitz wrote: Is there any need to validate anymore now that the volume configuration is gone? That's what was going to be validated correct? Bill Farner wrote: I would like to see an argument to specify which `ContainerType`s the scheduler should allow. This way the scheduler will not fatefully accept jobs and launch tasks that are not supported by the slave's command line and environment. Steve Niemitz wrote: Do you have any ideas on how to plumb that into the ConfigurationManager? Since it's a static class. I replied above with a suggestion. Copying here to save you from excessive clicking in reviewboard: Do you have any thoughts on how to get the setting propegated into the ConfigurationManager? We have some precedent for defining them as static fields in classes like this, but have started to backpedal on that as it is hostile to unit testing. Perhaps you can create a ConfigurationSettings class, to be injected into SchedulerThriftInterface and passed to ConfigurationManager? One thing to keep in mind - there are two primary entrypoints to ConfigurationManager - one for thrift, and the other for storage recovery. The thrift entrypoint tends to do validation and storage recovery mostly does backfilling of fields, to make sure that we continue to run jobs we have previously accepted. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68098 --- On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 8:20 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 14, 2015, 7:55 p.m., Bill Farner wrote: All nits this round. Last issue for me, as you point out, is validation/restriction of `ContainerType` in `ConfigurationManager. Doing a pass on test coverage in the meantime. Bill Farner wrote: Test coverage LGTM when supplemented with https://reviews.apache.org/r/29827/ Is there any need to validate anymore now that the volume configuration is gone? That's what was going to be validated correct? - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68098 --- On Jan. 14, 2015, 7:14 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 7:14 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29873: Fixing batched kill task filtering.
On Jan. 14, 2015, 7:21 p.m., Bill Farner wrote: src/main/python/apache/aurora/client/cli/context.py, line 214 https://reviews.apache.org/r/29873/diff/1/?file=820320#file820320line214 Seems like it would make sense for this to be the default behavior of `get_job_status`. I can't imagine why the default should be to fetch the active tasks plus some arbitrary history (since history is pruned) of dead instances. Meta: we should ~always do filtering like this server-side if possible. In this case, you should pass ACTIVE_STATES as a query parameter. `get_status_for_jobs` is also guilty of this, but would benefit from the proposd default behavior of `get_job_status` as well. Maxim Khutornenko wrote: The get_job_status is used by aurora job status to pull all available instances (including terminal ones). I was split between using a pre- and post- query filters and decided for the latter as a simpler one. The perf gain is quite neglible given the limited history we persist but I am happy to reconsider if you think it's worth it. I'm pretty strongly in favor of filtering with the query. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29873/#review68078 --- On Jan. 14, 2015, 1:16 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29873/ --- (Updated Jan. 14, 2015, 1:16 a.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-996 https://issues.apache.org/jira/browse/AURORA-996 Repository: aurora Description --- Fixed job status quering in client context to return active tasks where needed. Also, dropped active instance validation from both updater calls as it was hiding a legitimate feature: add new job instances with instance_spec. Diffs - src/main/python/apache/aurora/client/cli/context.py 93587c616afb0b7493a509197361cd76af2e2c97 src/main/python/apache/aurora/client/cli/jobs.py 508c9be556998e47bddcec8dee43f1595497b354 src/main/python/apache/aurora/client/cli/update.py a1617325f08ca252bdba38618aef141504ba7272 src/test/python/apache/aurora/client/cli/test_kill.py b475d737ede8ff0a669a9a9229196a76b43b46b6 src/test/python/apache/aurora/client/cli/test_restart.py a532ead256869c620e6bd96886ce9681b3423d0c Diff: https://reviews.apache.org/r/29873/diff/ Testing --- ./pants src/test/python:all vagrant@vagrant-ubuntu-trusty-64:~$ aurora job killall devcluster/www-data/prod/hello INFO] Checking status of devcluster/www-data/prod/hello INFO] No tasks to kill found for job devcluster/www-data/prod/hello Job killall succeeded Thanks, Maxim Khutornenko
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68125 --- Master (a350982) is red with this patch. ./build-support/jenkins/build.sh Installing /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/twitter.common.string-0.3.1-py2.7-nspkg.pth Running setup.py install for twitter.common.threading Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/twitter/__init__.py (namespace package) Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/twitter/common/__init__.py (namespace package) Installing /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/twitter.common.threading-0.3.1-py2.7-nspkg.pth Running setup.py install for twitter.common.util Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/twitter/__init__.py (namespace package) Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/twitter/common/__init__.py (namespace package) Installing /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/twitter.common.util-0.3.1-py2.7-nspkg.pth Found existing installation: setuptools 3.6 Uninstalling setuptools: Successfully uninstalled setuptools Successfully installed Markdown Pygments ansicolors cov-core coverage lockfile pantsbuild.pants pex psutil py pystache pytest pytest-cov python-daemon requests twitter.common.collections twitter.common.config twitter.common.confluence twitter.common.contextutil twitter.common.decorators twitter.common.dirutil twitter.common.lang twitter.common.log twitter.common.options twitter.common.process twitter.common.string twitter.common.threading twitter.common.util setuptools Cleaning up... Unable to detect suitable interpreter. Usage: pants build (options) [spec] (build args) pants build (options) [spec]... -- (build args) Options: --version show program's version number and exit --no-pantsrc Specifies that pantsrc files should be ignored. --log-exitLog an exit message on success or failure. -t CONN_TIMEOUT, --timeout=CONN_TIMEOUT Number of seconds to wait for http connections. -i INTERPRETERS, --interpreter=INTERPRETERS Constrain what Python interpreters to use. Uses Requirement format from pkg_resources, e.g. 'CPython=2.6,3' or 'PyPy'. By default, no constraints are used. Multiple constraints may be added. They will be ORed together. -v, --verbose Show verbose output. -f, --fastRun tests in a single chroot. Builds the specified Python target(s). Use ./pants goal for JVM and other targets. I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 8:20 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 8:20 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 14, 2015, 7:55 p.m., Bill Farner wrote: All nits this round. Last issue for me, as you point out, is validation/restriction of `ContainerType` in `ConfigurationManager. Doing a pass on test coverage in the meantime. Bill Farner wrote: Test coverage LGTM when supplemented with https://reviews.apache.org/r/29827/ Steve Niemitz wrote: Is there any need to validate anymore now that the volume configuration is gone? That's what was going to be validated correct? Bill Farner wrote: I would like to see an argument to specify which `ContainerType`s the scheduler should allow. This way the scheduler will not fatefully accept jobs and launch tasks that are not supported by the slave's command line and environment. Do you have any ideas on how to plumb that into the ConfigurationManager? Since it's a static class. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68098 --- On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 8:20 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
Re: Review Request 28920: Add support for docker containers to aurora
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/#review68112
---
Master (a350982) is red with this patch.
./build-support/jenkins/build.sh
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/test_builder.py,
line 88, in run
rv = self._run_tests([target], stdout, stderr)
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/test_builder.py,
line 316, in _run_tests
with self._test_runner(targets, stdout, stderr) as (pex, test_args):
File /usr/lib/python2.7/contextlib.py, line 17, in __enter__
return self.gen.next()
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/test_builder.py,
line 295, in _test_runner
builder = chroot.dump()
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/python_chroot.py,
line 191, in dump
conn_timeout=self._conn_timeout)
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/resolver.py,
line 100, in resolve_multi
platform=platform)
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pex/resolver.py,
line 110, in resolve
new_requirements.update(requires(highest_package, requirement))
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pex/resolver.py,
line 84, in requires
raise Untranslateable('Package %s is not translateable.' % package)
Untranslateable: Package
WheelPackage(u'https://pypi.python.org/packages/2.7/r/requests/requests-2.3.0-py2.py3-none-any.whl#md5=f2d850fd48fc10a93aa03d69b87b96b4')
is not translateable.
Usage:
pants build (options) [spec] (build args)
pants build (options) [spec]... -- (build args)
Options:
--version show program's version number and exit
--no-pantsrc Specifies that pantsrc files should be ignored.
--log-exitLog an exit message on success or failure.
-t CONN_TIMEOUT, --timeout=CONN_TIMEOUT
Number of seconds to wait for http connections.
-i INTERPRETERS, --interpreter=INTERPRETERS
Constrain what Python interpreters to use. Uses
Requirement format from pkg_resources, e.g.
'CPython=2.6,3' or 'PyPy'. By default, no
constraints are used. Multiple constraints may be
added. They will be ORed together.
-v, --verbose Show verbose output.
-f, --fastRun tests in a single chroot.
Builds the specified Python target(s). Use ./pants goal for JVM and other
targets.
I will refresh this build result if you post a review containing @ReviewBot
retry
- Aurora ReviewBot
On Jan. 14, 2015, 7:14 p.m., Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/
---
(Updated Jan. 14, 2015, 7:14 p.m.)
Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner.
Bugs: AURORA-633
https://issues.apache.org/jira/browse/AURORA-633
Repository: aurora
Description
---
This change adds support for launching docker containers through aurora.
These changes are based off of the discussion in
https://issues.apache.org/jira/browse/AURORA-633
As of now, a special thermos_executor.sh script is needed to launch the
executor inside docker containers. A sample aurora file is in
examples/jobs/docker.
In addition, mesos-slave must be run with `--containerizers=docker,mesos`,
the example upstart config in examples/vagrant/upstart has been updated to
reflect this.
More information is in subsequent review request comments.
Diffs
-
Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc
api/src/main/thrift/org/apache/aurora/gen/api.thrift
08ba1cdf88b712de22c26c04443079282db59ef9
config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5
docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea
docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7
examples/jobs/docker/hello_docker.aurora PRE-CREATION
examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd
examples/vagrant/provision-dev-cluster.sh
7af4b52a6876268a97630279221bb98d9b04efad
examples/vagrant/upstart/aurora-scheduler.conf
Re: Review Request 29873: Fixing batched kill task filtering.
On Jan. 14, 2015, 7:26 p.m., Bill Farner wrote: Also, dropped active instance validation from both updater calls as it was hiding a legitimate feature: add new job instances with instance_spec. I may have missed it - do unit tests verify this new behavior? Maxim Khutornenko wrote: The updater.py tests do. Since the functionality of validating instance spec is effectively removed, I don't see much benefit adding tests to validate the absent feature. I was probably too terse - you point out that you removed code because a legitimate use case was heretofore impossible. If that's the case, i'd like to make sure we have a test that prevents me from dropping those lines back in and regressing behavior. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29873/#review68082 --- On Jan. 14, 2015, 1:16 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29873/ --- (Updated Jan. 14, 2015, 1:16 a.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-996 https://issues.apache.org/jira/browse/AURORA-996 Repository: aurora Description --- Fixed job status quering in client context to return active tasks where needed. Also, dropped active instance validation from both updater calls as it was hiding a legitimate feature: add new job instances with instance_spec. Diffs - src/main/python/apache/aurora/client/cli/context.py 93587c616afb0b7493a509197361cd76af2e2c97 src/main/python/apache/aurora/client/cli/jobs.py 508c9be556998e47bddcec8dee43f1595497b354 src/main/python/apache/aurora/client/cli/update.py a1617325f08ca252bdba38618aef141504ba7272 src/test/python/apache/aurora/client/cli/test_kill.py b475d737ede8ff0a669a9a9229196a76b43b46b6 src/test/python/apache/aurora/client/cli/test_restart.py a532ead256869c620e6bd96886ce9681b3423d0c Diff: https://reviews.apache.org/r/29873/diff/ Testing --- ./pants src/test/python:all vagrant@vagrant-ubuntu-trusty-64:~$ aurora job killall devcluster/www-data/prod/hello INFO] Checking status of devcluster/www-data/prod/hello INFO] No tasks to kill found for job devcluster/www-data/prod/hello Job killall succeeded Thanks, Maxim Khutornenko
Re: Review Request 29866: Upgrade pants to 0.0.27
On Jan. 14, 2015, 12:49 a.m., Kevin Sweeney wrote: src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py, line 21 https://reviews.apache.org/r/29866/diff/3/?file=820203#file820203line21 same, this doesn't appear to be related to a pants upgrade Brian Wickman wrote: this is to fix the test failure brought up by reviewbot. unclear if it's related to the pants upgrade, but it should be fixed regardless. Brian Wickman wrote: would you prefer that this go into a separate review? Kevin Sweeney wrote: Yes, I prefer to keep commits small and focused (if I revert a commit titled upgrade pants I don't expect it to also break an unrelated test). Sorry if my comment made that unclear. Joshua Cohen wrote: If the failure is caused by the pants upgrade, then it should ship w/ this review. If it's just an oversight from the initial commit of the resource manager, then split it out? Well, reverting this change won't break that test -- the test passes on master and breaks with the pants upgrade. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29866/#review67988 --- On Jan. 14, 2015, 12:15 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29866/ --- (Updated Jan. 14, 2015, 12:15 a.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Repository: aurora Description --- Ports aurora repo to use pants 0.0.27 using the new pants new command processor. Diffs - .pantsversion 43b29618309bc6aa441848513d5dd3554a32c7c1 build-support/jenkins/build.sh 8de19cd865f3ae36c7ddc1aa0a8b453f95541b50 build-support/pants_requirements.txt 9a790f973cc7560980d43d5a1097338ff88a0397 build-support/python/make-pycharm-virtualenv 85e8137e8cab9765bd78ada6dedc9c60cc88e8a0 build-support/python/update-pants-requirements 523a3cb7e87146dc393d9ab09b23f2e3fb683b27 build-support/release/make-python-sdists 2f437d33928cc429a79aedb00247d472953fbb95 docs/developing-aurora-client.md a7253d2beba61f7aca94bab944f544b969f507fd examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd pants 40ddcf3a2c7359686ad326dd3d8ee37df55790e6 src/main/python/apache/aurora/executor/common/resource_manager.py bf69e7e60ae4dd857243d8b040a70daca008ea28 src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py e3c766f9b7bf989c0790281231957947d2cda52a src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 src/test/python/apache/aurora/executor/test_thermos_task_runner.py f8cf9fc7f0ed58be2bd2af5124e1a5df0dd2f00c Diff: https://reviews.apache.org/r/29866/diff/ Testing --- bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Brian Wickman
Re: Review Request 29866: Upgrade pants to 0.0.27
On Jan. 13, 2015, 4:49 p.m., Kevin Sweeney wrote: src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py, line 21 https://reviews.apache.org/r/29866/diff/3/?file=820203#file820203line21 same, this doesn't appear to be related to a pants upgrade Brian Wickman wrote: this is to fix the test failure brought up by reviewbot. unclear if it's related to the pants upgrade, but it should be fixed regardless. Brian Wickman wrote: would you prefer that this go into a separate review? Yes, I prefer to keep commits small and focused (if I revert a commit titled upgrade pants I don't expect it to also break an unrelated test). Sorry if my comment made that unclear. - Kevin --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29866/#review67988 --- On Jan. 13, 2015, 4:15 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29866/ --- (Updated Jan. 13, 2015, 4:15 p.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Repository: aurora Description --- Ports aurora repo to use pants 0.0.27 using the new pants new command processor. Diffs - .pantsversion 43b29618309bc6aa441848513d5dd3554a32c7c1 build-support/jenkins/build.sh 8de19cd865f3ae36c7ddc1aa0a8b453f95541b50 build-support/pants_requirements.txt 9a790f973cc7560980d43d5a1097338ff88a0397 build-support/python/make-pycharm-virtualenv 85e8137e8cab9765bd78ada6dedc9c60cc88e8a0 build-support/python/update-pants-requirements 523a3cb7e87146dc393d9ab09b23f2e3fb683b27 build-support/release/make-python-sdists 2f437d33928cc429a79aedb00247d472953fbb95 docs/developing-aurora-client.md a7253d2beba61f7aca94bab944f544b969f507fd examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd pants 40ddcf3a2c7359686ad326dd3d8ee37df55790e6 src/main/python/apache/aurora/executor/common/resource_manager.py bf69e7e60ae4dd857243d8b040a70daca008ea28 src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py e3c766f9b7bf989c0790281231957947d2cda52a src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 src/test/python/apache/aurora/executor/test_thermos_task_runner.py f8cf9fc7f0ed58be2bd2af5124e1a5df0dd2f00c Diff: https://reviews.apache.org/r/29866/diff/ Testing --- bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Brian Wickman
Re: Review Request 29866: Upgrade pants to 0.0.27
On Jan. 14, 2015, 12:49 a.m., Kevin Sweeney wrote: src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py, line 21 https://reviews.apache.org/r/29866/diff/3/?file=820203#file820203line21 same, this doesn't appear to be related to a pants upgrade Brian Wickman wrote: this is to fix the test failure brought up by reviewbot. unclear if it's related to the pants upgrade, but it should be fixed regardless. Brian Wickman wrote: would you prefer that this go into a separate review? Kevin Sweeney wrote: Yes, I prefer to keep commits small and focused (if I revert a commit titled upgrade pants I don't expect it to also break an unrelated test). Sorry if my comment made that unclear. If the failure is caused by the pants upgrade, then it should ship w/ this review. If it's just an oversight from the initial commit of the resource manager, then split it out? - Joshua --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29866/#review67988 --- On Jan. 14, 2015, 12:15 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29866/ --- (Updated Jan. 14, 2015, 12:15 a.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Repository: aurora Description --- Ports aurora repo to use pants 0.0.27 using the new pants new command processor. Diffs - .pantsversion 43b29618309bc6aa441848513d5dd3554a32c7c1 build-support/jenkins/build.sh 8de19cd865f3ae36c7ddc1aa0a8b453f95541b50 build-support/pants_requirements.txt 9a790f973cc7560980d43d5a1097338ff88a0397 build-support/python/make-pycharm-virtualenv 85e8137e8cab9765bd78ada6dedc9c60cc88e8a0 build-support/python/update-pants-requirements 523a3cb7e87146dc393d9ab09b23f2e3fb683b27 build-support/release/make-python-sdists 2f437d33928cc429a79aedb00247d472953fbb95 docs/developing-aurora-client.md a7253d2beba61f7aca94bab944f544b969f507fd examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd pants 40ddcf3a2c7359686ad326dd3d8ee37df55790e6 src/main/python/apache/aurora/executor/common/resource_manager.py bf69e7e60ae4dd857243d8b040a70daca008ea28 src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py e3c766f9b7bf989c0790281231957947d2cda52a src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 src/test/python/apache/aurora/executor/test_thermos_task_runner.py f8cf9fc7f0ed58be2bd2af5124e1a5df0dd2f00c Diff: https://reviews.apache.org/r/29866/diff/ Testing --- bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Brian Wickman
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68132 --- Ship it! Master (a350982) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 8:20 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/#review68155
---
Master (a350982) is red with this patch.
./build-support/jenkins/build.sh
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/test_builder.py,
line 88, in run
rv = self._run_tests([target], stdout, stderr)
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/test_builder.py,
line 316, in _run_tests
with self._test_runner(targets, stdout, stderr) as (pex, test_args):
File /usr/lib/python2.7/contextlib.py, line 17, in __enter__
return self.gen.next()
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/test_builder.py,
line 295, in _test_runner
builder = chroot.dump()
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/python_chroot.py,
line 191, in dump
conn_timeout=self._conn_timeout)
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/resolver.py,
line 100, in resolve_multi
platform=platform)
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pex/resolver.py,
line 110, in resolve
new_requirements.update(requires(highest_package, requirement))
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pex/resolver.py,
line 84, in requires
raise Untranslateable('Package %s is not translateable.' % package)
Untranslateable: Package
SourcePackage(u'https://pypi.python.org/packages/source/t/twitter.common.dirutil/twitter.common.dirutil-0.3.2.tar.gz#md5=8f13bf805cd55dcdfe0d4dc8ea507982')
is not translateable.
Usage:
pants build (options) [spec] (build args)
pants build (options) [spec]... -- (build args)
Options:
--version show program's version number and exit
--no-pantsrc Specifies that pantsrc files should be ignored.
--log-exitLog an exit message on success or failure.
-t CONN_TIMEOUT, --timeout=CONN_TIMEOUT
Number of seconds to wait for http connections.
-i INTERPRETERS, --interpreter=INTERPRETERS
Constrain what Python interpreters to use. Uses
Requirement format from pkg_resources, e.g.
'CPython=2.6,3' or 'PyPy'. By default, no
constraints are used. Multiple constraints may be
added. They will be ORed together.
-v, --verbose Show verbose output.
-f, --fastRun tests in a single chroot.
Builds the specified Python target(s). Use ./pants goal for JVM and other
targets.
I will refresh this build result if you post a review containing @ReviewBot
retry
- Aurora ReviewBot
On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/
---
(Updated Jan. 14, 2015, 8:20 p.m.)
Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner.
Bugs: AURORA-633
https://issues.apache.org/jira/browse/AURORA-633
Repository: aurora
Description
---
This change adds support for launching docker containers through aurora.
These changes are based off of the discussion in
https://issues.apache.org/jira/browse/AURORA-633
As of now, a special thermos_executor.sh script is needed to launch the
executor inside docker containers. A sample aurora file is in
examples/jobs/docker.
In addition, mesos-slave must be run with `--containerizers=docker,mesos`,
the example upstart config in examples/vagrant/upstart has been updated to
reflect this.
More information is in subsequent review request comments.
Diffs
-
Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc
api/src/main/thrift/org/apache/aurora/gen/api.thrift
08ba1cdf88b712de22c26c04443079282db59ef9
config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5
docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea
docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7
examples/jobs/docker/hello_docker.aurora PRE-CREATION
examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd
examples/vagrant/provision-dev-cluster.sh
7af4b52a6876268a97630279221bb98d9b04efad
examples/vagrant/upstart/aurora-scheduler.conf
Review Request 29901: De-flakify resource_manager_integration test
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29901/ --- Review request for Aurora and Kevin Sweeney. Repository: aurora Description --- De-flakify ResourceManager integration test. Diffs - src/main/python/apache/aurora/executor/common/resource_manager.py bf69e7e60ae4dd857243d8b040a70daca008ea28 src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py e3c766f9b7bf989c0790281231957947d2cda52a Diff: https://reviews.apache.org/r/29901/diff/ Testing --- ./pants goal test --no-test-pytest-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 29866: Upgrade pants to 0.0.27
On Jan. 14, 2015, 12:49 a.m., Kevin Sweeney wrote: src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py, line 21 https://reviews.apache.org/r/29866/diff/3/?file=820203#file820203line21 same, this doesn't appear to be related to a pants upgrade Brian Wickman wrote: this is to fix the test failure brought up by reviewbot. unclear if it's related to the pants upgrade, but it should be fixed regardless. would you prefer that this go into a separate review? - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29866/#review67988 --- On Jan. 14, 2015, 12:15 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29866/ --- (Updated Jan. 14, 2015, 12:15 a.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Repository: aurora Description --- Ports aurora repo to use pants 0.0.27 using the new pants new command processor. Diffs - .pantsversion 43b29618309bc6aa441848513d5dd3554a32c7c1 build-support/jenkins/build.sh 8de19cd865f3ae36c7ddc1aa0a8b453f95541b50 build-support/pants_requirements.txt 9a790f973cc7560980d43d5a1097338ff88a0397 build-support/python/make-pycharm-virtualenv 85e8137e8cab9765bd78ada6dedc9c60cc88e8a0 build-support/python/update-pants-requirements 523a3cb7e87146dc393d9ab09b23f2e3fb683b27 build-support/release/make-python-sdists 2f437d33928cc429a79aedb00247d472953fbb95 docs/developing-aurora-client.md a7253d2beba61f7aca94bab944f544b969f507fd examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd pants 40ddcf3a2c7359686ad326dd3d8ee37df55790e6 src/main/python/apache/aurora/executor/common/resource_manager.py bf69e7e60ae4dd857243d8b040a70daca008ea28 src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py e3c766f9b7bf989c0790281231957947d2cda52a src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 src/test/python/apache/aurora/executor/test_thermos_task_runner.py f8cf9fc7f0ed58be2bd2af5124e1a5df0dd2f00c Diff: https://reviews.apache.org/r/29866/diff/ Testing --- bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Brian Wickman
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68185 --- Master (a350982) is red with this patch. ./build-support/jenkins/build.sh :licenseTest UP-TO-DATE :license UP-TO-DATE :pmdMain :test :jacocoTestReport Coverage report generated: file:///x1/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/jacoco/test/html/index.html :analyzeReport Instruction coverage of 0.8935905559858246 exceeds minimum coverage of 0.89. Branch coverage of 0.8402275077559462 exceeds minimum coverage of 0.835. :check :build :api:assemble :api:compileTestJava UP-TO-DATE :api:processTestResources UP-TO-DATE :api:testClasses UP-TO-DATE :api:test UP-TO-DATE :api:check UP-TO-DATE :api:build :buildSrc:compileJava UP-TO-DATE :buildSrc:processResources UP-TO-DATE :buildSrc:classes UP-TO-DATE :buildSrc:jar :buildSrc:assemble :buildSrc:compileTestJava UP-TO-DATE :buildSrc:processTestResources UP-TO-DATE :buildSrc:testClasses UP-TO-DATE :buildSrc:test UP-TO-DATE :buildSrc:check UP-TO-DATE :buildSrc:build BUILD SUCCESSFUL Total time: 3 mins 0.215 secs + export PIP_DEFAULT_TIMEOUT=60 + PIP_DEFAULT_TIMEOUT=60 + mkdir -p third_party + pip install -d third_party -r /dev/fd/63 ++ grep -v mesos.native 3rdparty/python/requirements.txt ./build-support/jenkins/build.sh: line 28: pip: command not found grep: write error: Broken pipe I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 8:20 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js
Re: Review Request 29862: Remove documentation for nonexistent pystachio feature.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29862/#review68191 --- Ship it! Ship It! - Brian Wickman On Jan. 13, 2015, 7:45 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29862/ --- (Updated Jan. 13, 2015, 7:45 p.m.) Review request for Aurora and Brian Wickman. Bugs: AURORA-1008 https://issues.apache.org/jira/browse/AURORA-1008 Repository: aurora Description --- Remove documentation for nonexistent pystachio feature. Diffs - docs/configuration-tutorial.md b64dc14649939083821ac682b08a5d3eb6e8ae41 Diff: https://reviews.apache.org/r/29862/diff/ Testing --- Thanks, Kevin Sweeney
Re: Review Request 29915: Prevent NPE in Preemptor.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29915/ --- (Updated Jan. 14, 2015, 6:26 p.m.) Review request for Aurora. Repository: aurora Description --- It is possible for CachedClusterState to add a null SlaveID to the multimap. This null key can propagate to the Preemptor and cause the following NPE: W0114 20:57:59.565 THREAD149 org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.schedule: Task scheduling unexpectedly failed, will be retried java.lang.NullPointerException at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) at com.google.common.collect.ImmutableCollection$ArrayBasedBuilder.add(ImmutableCollection.java:339) at com.google.common.collect.ImmutableSet$Builder.add(ImmutableSet.java:480) at com.google.common.collect.ImmutableSet$Builder.add(ImmutableSet.java:456) at com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:282) at com.google.common.collect.ImmutableCollection$ArrayBasedBuilder.addAll(ImmutableCollection.java:360) at com.google.common.collect.ImmutableSet$Builder.addAll(ImmutableSet.java:508) at org.apache.aurora.scheduler.async.preemptor.PreemptorImpl.findPreemptionSlotFor(PreemptorImpl.java:321) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.maybePreemptFor(TaskScheduler.java:249) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.scheduleTask(TaskScheduler.java:220) at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl$3.apply(TaskScheduler.java:192) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl$3.apply(TaskScheduler.java:189) at org.apache.aurora.scheduler.storage.log.LogStorage$24.apply(LogStorage.java:608) at org.apache.aurora.scheduler.storage.log.LogStorage$24.apply(LogStorage.java:605) at org.apache.aurora.scheduler.storage.mem.MemStorage$3.apply(MemStorage.java:147) at org.apache.aurora.scheduler.storage.mem.MemStorage$3.apply(MemStorage.java:144) at org.apache.aurora.scheduler.storage.db.DbStorage.write(DbStorage.java:137) at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:101) at org.apache.aurora.scheduler.storage.mem.MemStorage.write(MemStorage.java:144) at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87) at org.apache.aurora.scheduler.storage.log.LogStorage.doInTransaction(LogStorage.java:605) at org.apache.aurora.scheduler.storage.log.LogStorage.write(LogStorage.java:638) at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.write(CallOrderEnforcingStorage.java:122) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.schedule(TaskScheduler.java:189) at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87) at org.apache.aurora.scheduler.async.TaskGroups$1.schedule(TaskGroups.java:136) at org.apache.aurora.scheduler.async.TaskGroups$2.run(TaskGroups.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) This patch prevents the addition of the null key into the multimap and instead throws an `IllegalStateException`. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/async/preemptor/CachedClusterState.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/async/preemptor/CachedClusterStateTest.java PRE-CREATION Diff: https://reviews.apache.org/r/29915/diff/ Testing --- ./gradlew build -Pq Thanks, Zameer Manji
Re: Review Request 29915: Prevent NPE in Preemptor.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29915/#review68195 --- Ship it! Master (a350982) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 15, 2015, 2:27 a.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29915/ --- (Updated Jan. 15, 2015, 2:27 a.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-1016 https://issues.apache.org/jira/browse/AURORA-1016 Repository: aurora Description --- It is possible for CachedClusterState to add a null SlaveID to the multimap. This null key can propagate to the Preemptor and cause the following NPE: W0114 20:57:59.565 THREAD149 org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.schedule: Task scheduling unexpectedly failed, will be retried java.lang.NullPointerException at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) at com.google.common.collect.ImmutableCollection$ArrayBasedBuilder.add(ImmutableCollection.java:339) at com.google.common.collect.ImmutableSet$Builder.add(ImmutableSet.java:480) at com.google.common.collect.ImmutableSet$Builder.add(ImmutableSet.java:456) at com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:282) at com.google.common.collect.ImmutableCollection$ArrayBasedBuilder.addAll(ImmutableCollection.java:360) at com.google.common.collect.ImmutableSet$Builder.addAll(ImmutableSet.java:508) at org.apache.aurora.scheduler.async.preemptor.PreemptorImpl.findPreemptionSlotFor(PreemptorImpl.java:321) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.maybePreemptFor(TaskScheduler.java:249) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.scheduleTask(TaskScheduler.java:220) at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl$3.apply(TaskScheduler.java:192) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl$3.apply(TaskScheduler.java:189) at org.apache.aurora.scheduler.storage.log.LogStorage$24.apply(LogStorage.java:608) at org.apache.aurora.scheduler.storage.log.LogStorage$24.apply(LogStorage.java:605) at org.apache.aurora.scheduler.storage.mem.MemStorage$3.apply(MemStorage.java:147) at org.apache.aurora.scheduler.storage.mem.MemStorage$3.apply(MemStorage.java:144) at org.apache.aurora.scheduler.storage.db.DbStorage.write(DbStorage.java:137) at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:101) at org.apache.aurora.scheduler.storage.mem.MemStorage.write(MemStorage.java:144) at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87) at org.apache.aurora.scheduler.storage.log.LogStorage.doInTransaction(LogStorage.java:605) at org.apache.aurora.scheduler.storage.log.LogStorage.write(LogStorage.java:638) at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.write(CallOrderEnforcingStorage.java:122) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.schedule(TaskScheduler.java:189) at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87) at org.apache.aurora.scheduler.async.TaskGroups$1.schedule(TaskGroups.java:136) at org.apache.aurora.scheduler.async.TaskGroups$2.run(TaskGroups.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) This patch prevents the addition of the null key into the multimap and instead throws an `IllegalStateException`. Diffs - src/main/java/org/apache/aurora/scheduler/async/preemptor/CachedClusterState.java PRE-CREATION
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68203 --- Master (a350982) is red with this patch. ./build-support/jenkins/build.sh :licenseTest UP-TO-DATE :license UP-TO-DATE :pmdMain :test :jacocoTestReport Coverage report generated: file:///x1/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/jacoco/test/html/index.html :analyzeReport Instruction coverage of 0.893543617545589 exceeds minimum coverage of 0.89. Branch coverage of 0.8391933815925543 exceeds minimum coverage of 0.835. :check :build :api:assemble :api:compileTestJava UP-TO-DATE :api:processTestResources UP-TO-DATE :api:testClasses UP-TO-DATE :api:test UP-TO-DATE :api:check UP-TO-DATE :api:build :buildSrc:compileJava UP-TO-DATE :buildSrc:processResources UP-TO-DATE :buildSrc:classes UP-TO-DATE :buildSrc:jar :buildSrc:assemble :buildSrc:compileTestJava UP-TO-DATE :buildSrc:processTestResources UP-TO-DATE :buildSrc:testClasses UP-TO-DATE :buildSrc:test UP-TO-DATE :buildSrc:check UP-TO-DATE :buildSrc:build BUILD SUCCESSFUL Total time: 2 mins 59.342 secs + export PIP_DEFAULT_TIMEOUT=60 + PIP_DEFAULT_TIMEOUT=60 + mkdir -p third_party + pip install -d third_party -r /dev/fd/63 ++ grep -v mesos.native 3rdparty/python/requirements.txt ./build-support/jenkins/build.sh: line 28: pip: command not found grep: write error: Broken pipe I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 8:20 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js
Re: Review Request 28920: Add support for docker containers to aurora
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/#review68196
---
Master (a350982) is red with this patch.
./build-support/jenkins/build.sh
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/test_builder.py,
line 88, in run
rv = self._run_tests([target], stdout, stderr)
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/test_builder.py,
line 316, in _run_tests
with self._test_runner(targets, stdout, stderr) as (pex, test_args):
File /usr/lib/python2.7/contextlib.py, line 17, in __enter__
return self.gen.next()
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/test_builder.py,
line 295, in _test_runner
builder = chroot.dump()
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/python_chroot.py,
line 191, in dump
conn_timeout=self._conn_timeout)
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/resolver.py,
line 100, in resolve_multi
platform=platform)
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pex/resolver.py,
line 110, in resolve
new_requirements.update(requires(highest_package, requirement))
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pex/resolver.py,
line 84, in requires
raise Untranslateable('Package %s is not translateable.' % package)
Untranslateable: Package
SourcePackage(u'https://pypi.python.org/packages/source/t/thrift/thrift-0.9.1.tar.gz#md5=8989a8a96b0e3a3380cfb89c44e172a6')
is not translateable.
Usage:
pants build (options) [spec] (build args)
pants build (options) [spec]... -- (build args)
Options:
--version show program's version number and exit
--no-pantsrc Specifies that pantsrc files should be ignored.
--log-exitLog an exit message on success or failure.
-t CONN_TIMEOUT, --timeout=CONN_TIMEOUT
Number of seconds to wait for http connections.
-i INTERPRETERS, --interpreter=INTERPRETERS
Constrain what Python interpreters to use. Uses
Requirement format from pkg_resources, e.g.
'CPython=2.6,3' or 'PyPy'. By default, no
constraints are used. Multiple constraints may be
added. They will be ORed together.
-v, --verbose Show verbose output.
-f, --fastRun tests in a single chroot.
Builds the specified Python target(s). Use ./pants goal for JVM and other
targets.
I will refresh this build result if you post a review containing @ReviewBot
retry
- Aurora ReviewBot
On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/
---
(Updated Jan. 14, 2015, 8:20 p.m.)
Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner.
Bugs: AURORA-633
https://issues.apache.org/jira/browse/AURORA-633
Repository: aurora
Description
---
This change adds support for launching docker containers through aurora.
These changes are based off of the discussion in
https://issues.apache.org/jira/browse/AURORA-633
As of now, a special thermos_executor.sh script is needed to launch the
executor inside docker containers. A sample aurora file is in
examples/jobs/docker.
In addition, mesos-slave must be run with `--containerizers=docker,mesos`,
the example upstart config in examples/vagrant/upstart has been updated to
reflect this.
More information is in subsequent review request comments.
Diffs
-
Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc
api/src/main/thrift/org/apache/aurora/gen/api.thrift
08ba1cdf88b712de22c26c04443079282db59ef9
config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5
docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea
docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7
examples/jobs/docker/hello_docker.aurora PRE-CREATION
examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd
examples/vagrant/provision-dev-cluster.sh
7af4b52a6876268a97630279221bb98d9b04efad
examples/vagrant/upstart/aurora-scheduler.conf
788ec254270bca074dae91829c7f4fccdc8f8bb0
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68210 --- Ship it! Master (a350982) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 8:20 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68200 --- Ship it! Master (a350982) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 8:20 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29915: Prevent NPE in Preemptor.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29915/ --- (Updated Jan. 14, 2015, 6:27 p.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-1016 https://issues.apache.org/jira/browse/AURORA-1016 Repository: aurora Description --- It is possible for CachedClusterState to add a null SlaveID to the multimap. This null key can propagate to the Preemptor and cause the following NPE: W0114 20:57:59.565 THREAD149 org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.schedule: Task scheduling unexpectedly failed, will be retried java.lang.NullPointerException at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) at com.google.common.collect.ImmutableCollection$ArrayBasedBuilder.add(ImmutableCollection.java:339) at com.google.common.collect.ImmutableSet$Builder.add(ImmutableSet.java:480) at com.google.common.collect.ImmutableSet$Builder.add(ImmutableSet.java:456) at com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:282) at com.google.common.collect.ImmutableCollection$ArrayBasedBuilder.addAll(ImmutableCollection.java:360) at com.google.common.collect.ImmutableSet$Builder.addAll(ImmutableSet.java:508) at org.apache.aurora.scheduler.async.preemptor.PreemptorImpl.findPreemptionSlotFor(PreemptorImpl.java:321) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.maybePreemptFor(TaskScheduler.java:249) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.scheduleTask(TaskScheduler.java:220) at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl$3.apply(TaskScheduler.java:192) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl$3.apply(TaskScheduler.java:189) at org.apache.aurora.scheduler.storage.log.LogStorage$24.apply(LogStorage.java:608) at org.apache.aurora.scheduler.storage.log.LogStorage$24.apply(LogStorage.java:605) at org.apache.aurora.scheduler.storage.mem.MemStorage$3.apply(MemStorage.java:147) at org.apache.aurora.scheduler.storage.mem.MemStorage$3.apply(MemStorage.java:144) at org.apache.aurora.scheduler.storage.db.DbStorage.write(DbStorage.java:137) at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:101) at org.apache.aurora.scheduler.storage.mem.MemStorage.write(MemStorage.java:144) at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87) at org.apache.aurora.scheduler.storage.log.LogStorage.doInTransaction(LogStorage.java:605) at org.apache.aurora.scheduler.storage.log.LogStorage.write(LogStorage.java:638) at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.write(CallOrderEnforcingStorage.java:122) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.schedule(TaskScheduler.java:189) at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87) at org.apache.aurora.scheduler.async.TaskGroups$1.schedule(TaskGroups.java:136) at org.apache.aurora.scheduler.async.TaskGroups$2.run(TaskGroups.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) This patch prevents the addition of the null key into the multimap and instead throws an `IllegalStateException`. Diffs - src/main/java/org/apache/aurora/scheduler/async/preemptor/CachedClusterState.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/async/preemptor/CachedClusterStateTest.java PRE-CREATION Diff: https://reviews.apache.org/r/29915/diff/ Testing --- ./gradlew build -Pq Thanks, Zameer Manji
Review Request 29915: Prevent NPE in Preemptor.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29915/ --- Review request for Aurora. Repository: aurora Description --- It is possible for CachedClusterState to add a null SlaveID to the multimap. This null key can propagate to the Preemptor and cause the following NPE: W0114 20:57:59.565 THREAD149 org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.schedule: Task scheduling unexpectedly failed, will be retried java.lang.NullPointerException at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) at com.google.common.collect.ImmutableCollection$ArrayBasedBuilder.add(ImmutableCollection.java:339) at com.google.common.collect.ImmutableSet$Builder.add(ImmutableSet.java:480) at com.google.common.collect.ImmutableSet$Builder.add(ImmutableSet.java:456) at com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:282) at com.google.common.collect.ImmutableCollection$ArrayBasedBuilder.addAll(ImmutableCollection.java:360) at com.google.common.collect.ImmutableSet$Builder.addAll(ImmutableSet.java:508) at org.apache.aurora.scheduler.async.preemptor.PreemptorImpl.findPreemptionSlotFor(PreemptorImpl.java:321) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.maybePreemptFor(TaskScheduler.java:249) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.scheduleTask(TaskScheduler.java:220) at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl$3.apply(TaskScheduler.java:192) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl$3.apply(TaskScheduler.java:189) at org.apache.aurora.scheduler.storage.log.LogStorage$24.apply(LogStorage.java:608) at org.apache.aurora.scheduler.storage.log.LogStorage$24.apply(LogStorage.java:605) at org.apache.aurora.scheduler.storage.mem.MemStorage$3.apply(MemStorage.java:147) at org.apache.aurora.scheduler.storage.mem.MemStorage$3.apply(MemStorage.java:144) at org.apache.aurora.scheduler.storage.db.DbStorage.write(DbStorage.java:137) at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:101) at org.apache.aurora.scheduler.storage.mem.MemStorage.write(MemStorage.java:144) at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87) at org.apache.aurora.scheduler.storage.log.LogStorage.doInTransaction(LogStorage.java:605) at org.apache.aurora.scheduler.storage.log.LogStorage.write(LogStorage.java:638) at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.write(CallOrderEnforcingStorage.java:122) at org.apache.aurora.scheduler.async.TaskScheduler$TaskSchedulerImpl.schedule(TaskScheduler.java:189) at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87) at org.apache.aurora.scheduler.async.TaskGroups$1.schedule(TaskGroups.java:136) at org.apache.aurora.scheduler.async.TaskGroups$2.run(TaskGroups.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) This patch prevents the addition of the null key into the multimap and instead throws an `IllegalStateException`. Diffs - src/main/java/org/apache/aurora/scheduler/async/preemptor/CachedClusterState.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/async/preemptor/CachedClusterStateTest.java PRE-CREATION Diff: https://reviews.apache.org/r/29915/diff/ Testing --- ./gradlew build -Pq Thanks, Zameer Manji
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68213 --- Ship it! Master (a350982) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 8:20 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/#review68219
---
Master (a350982) is red with this patch.
./build-support/jenkins/build.sh
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/python_builder.py,
line 43, in build
debug=debug).run()
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/test_builder.py,
line 88, in run
rv = self._run_tests([target], stdout, stderr)
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/test_builder.py,
line 316, in _run_tests
with self._test_runner(targets, stdout, stderr) as (pex, test_args):
File /usr/lib/python2.7/contextlib.py, line 17, in __enter__
return self.gen.next()
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/test_builder.py,
line 295, in _test_runner
builder = chroot.dump()
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/python_chroot.py,
line 191, in dump
conn_timeout=self._conn_timeout)
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pants/backend/python/resolver.py,
line 100, in resolve_multi
platform=platform)
File
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/pex/resolver.py,
line 101, in resolve
raise Unsatisfiable('Cannot satisfy requirements: %s' %
requirement_set[requirement.key])
Unsatisfiable: Cannot satisfy requirements: [Requirement.parse('py=1.4.22')]
Usage:
pants build (options) [spec] (build args)
pants build (options) [spec]... -- (build args)
Options:
--version show program's version number and exit
--no-pantsrc Specifies that pantsrc files should be ignored.
--log-exitLog an exit message on success or failure.
-t CONN_TIMEOUT, --timeout=CONN_TIMEOUT
Number of seconds to wait for http connections.
-i INTERPRETERS, --interpreter=INTERPRETERS
Constrain what Python interpreters to use. Uses
Requirement format from pkg_resources, e.g.
'CPython=2.6,3' or 'PyPy'. By default, no
constraints are used. Multiple constraints may be
added. They will be ORed together.
-v, --verbose Show verbose output.
-f, --fastRun tests in a single chroot.
Builds the specified Python target(s). Use ./pants goal for JVM and other
targets.
I will refresh this build result if you post a review containing @ReviewBot
retry
- Aurora ReviewBot
On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/
---
(Updated Jan. 14, 2015, 8:20 p.m.)
Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner.
Bugs: AURORA-633
https://issues.apache.org/jira/browse/AURORA-633
Repository: aurora
Description
---
This change adds support for launching docker containers through aurora.
These changes are based off of the discussion in
https://issues.apache.org/jira/browse/AURORA-633
As of now, a special thermos_executor.sh script is needed to launch the
executor inside docker containers. A sample aurora file is in
examples/jobs/docker.
In addition, mesos-slave must be run with `--containerizers=docker,mesos`,
the example upstart config in examples/vagrant/upstart has been updated to
reflect this.
More information is in subsequent review request comments.
Diffs
-
Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc
api/src/main/thrift/org/apache/aurora/gen/api.thrift
08ba1cdf88b712de22c26c04443079282db59ef9
config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5
docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea
docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7
examples/jobs/docker/hello_docker.aurora PRE-CREATION
examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd
examples/vagrant/provision-dev-cluster.sh
7af4b52a6876268a97630279221bb98d9b04efad
examples/vagrant/upstart/aurora-scheduler.conf
788ec254270bca074dae91829c7f4fccdc8f8bb0
examples/vagrant/upstart/mesos-slave.conf
512ce7ecf34042ed68dda55efb2dd0415f8469db
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 14, 2015, 2:28 a.m., Kevin Sweeney wrote: examples/jobs/docker/hello_docker.aurora, line 5 https://reviews.apache.org/r/28920/diff/13/?file=818518#file818518line5 since we're supposedly running in the python2.7 container could we show our version? Bonus points, use a python3 container to prove that this is different from the host version? If it were a python 3 container it would also need python 2.7 in order to run the executor. This limitation is called out in the docs in deploying aurora scheduler under Considerations for running jobs in docker containers however I'll add it as a note in the configuration reference as well. On Jan. 14, 2015, 2:28 a.m., Kevin Sweeney wrote: examples/vagrant/aurorabuild.sh, line 74 https://reviews.apache.org/r/28920/diff/13/?file=818519#file818519line74 is this wrapper necessary anymore? Nope, we can get rid of it. On Jan. 14, 2015, 2:28 a.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, lines 297-298 https://reviews.apache.org/r/28920/diff/13/?file=818527#file818527line297 How would you feel about removing these suffixes from the scheduler and letting the user specify an executor-per-container-type? I can see a deployment scenario where the executor is baked into the docker containers. I'll add a TODO here, but for the intial release here I don't think it's required to support that use case. On Jan. 14, 2015, 2:28 a.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line 328 https://reviews.apache.org/r/28920/diff/13/?file=818527#file818527line328 why the wrap/unwrap combination here? Thrift - Protobuf conversion. Is there a better way to do it? The ContainerInfo comment above will be the same. On Jan. 14, 2015, 2:28 a.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line 336 https://reviews.apache.org/r/28920/diff/13/?file=818527#file818527line336 Can the thermos observer detect container checkpoints any other way? IMO that would be preferable to plumbing it in on the scheduler-side. Not without big changes to the observer. This can go away if https://issues.apache.org/jira/browse/AURORA-708 is implemented. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68013 --- On Jan. 13, 2015, 1:11 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 13, 2015, 1:11 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68062 --- Third time's the charm? I feel like I'm trying to shift without using a clutch... @ReviewBot retry - Steve Niemitz On Jan. 14, 2015, 4:50 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 4:50 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
