Re: Review Request 28920: Add support for docker containers to aurora
On Dec. 13, 2014, 1:59 a.m., Jay Buffington wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line 279 https://reviews.apache.org/r/28920/diff/2/?file=789365#file789365line279 I would do away with this TaskConfig hasProcesses field. You should just use config.isSetExecutorConfig() (and of course not set executor config in the python client when there are no processes.) Steve Niemitz wrote: I'm going to remove support for running the container directly w/o the executor for now. Without the job registering with the observer, aurora thinks the job is lost and will GC it or abandon it. We can add this as a to-do in the future. I'm +1 on this. I think running without the executor is of dubious value anyway. I hadn't considered the issue with the GC. Thanks for pointing this out! - Jay --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review65000 --- On Dec. 26, 2014, 9:05 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Dec. 26, 2014, 9:05 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample script is in examples/jobs/docker, as well as an example aurora file. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. The thermos root path defaults to /var/run/thermos, however if a different path is used, it must be passed to the scheduler via `--thermos_observer_root=some path` Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ead9d28100673440168a32d114ecaa15874978a6 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 5, 2015, 8:41 p.m., Joshua Cohen wrote:
This is 99% just style nits. Unfortunately our Java styleguide isn't
published, but I'm working on rectifying that!
Thanks for all the feedback, new patch set coming up!
On Jan. 5, 2015, 8:41 p.m., Joshua Cohen wrote:
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java, lines
223-224
https://reviews.apache.org/r/28920/diff/4/?file=801606#file801606line223
Do we need to plumb the wrapper path all the way through to the
CommandUtil? What if instead of passing both along we figure out the
correct path to use here at start up and passed that along directly?
I.e. we could do something like...
OptionalString executorPath =
Optional.of(THERMOS_EXECUTOR_PATH.get()).or(Optional.of(THERMOS_EXECUTOR_WRAPPER_PATH.get()));
if (!executorPath.isPresent()) {
throw new IllegalStateException(...);
}
bind(ExecutorSettings.class).toInstance(new ExecutorSettings(
executorPath.get(),
...));
We need both paths in the ExecutorSettings so they can both get added to the
CommandInfo as resources (for docker).
On Jan. 5, 2015, 8:41 p.m., Joshua Cohen wrote:
src/test/python/apache/aurora/executor/test_thermos_executor.py, line 193
https://reviews.apache.org/r/28920/diff/4/?file=801621#file801621line193
This seems like a change in semantics... the sandbox_provider
previously was expected to be a factory function that returned the sandbox,
now you're passing in the sandbox itself? Why the change?
The UserOverrideDirectorySandboxProvider has a constructor which takes
arguments, so the provider can no longer be parameterless. I feel like this is
more true to a factory pattern anyways, instead of passing in a factory
factory we're now just passing in a factory.
- Steve
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/#review66702
---
On Jan. 5, 2015, 8:25 p.m., Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/
---
(Updated Jan. 5, 2015, 8:25 p.m.)
Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner.
Bugs: AURORA-633
https://issues.apache.org/jira/browse/AURORA-633
Repository: aurora
Description
---
This change adds support for launching docker containers through aurora.
These changes are based off of the discussion in
https://issues.apache.org/jira/browse/AURORA-633
As of now, a special thermos_executor.sh script is needed to launch the
executor inside docker containers. A sample aurora file is in
examples/jobs/docker.
In addition, mesos-slave must be run with `--containerizers=docker,mesos`,
the example upstart config in examples/vagrant/upstart has been updated to
reflect this.
More information is in subsequent review request comments.
Diffs
-
Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc
api/src/main/thrift/org/apache/aurora/gen/api.thrift
5665c69cd7b49c3fd7345074c9f16a3b224496ab
docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea
docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7
examples/jobs/docker/hello_docker.aurora PRE-CREATION
examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262
examples/vagrant/upstart/mesos-slave.conf
512ce7ecf34042ed68dda55efb2dd0415f8469db
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java
72c7545e7f16549f6a9ccb5fb74a06f154a7ea94
src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java
5226e3d1b303b1773a057078f2911c5ec2aa97f5
src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java
ead9d28100673440168a32d114ecaa15874978a6
src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java
d885b224ec5a1d529347d84e03ba98ab6734a126
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
5bf283062c9d119ff91ed45da8b236e36d0fc9aa
src/main/python/apache/aurora/config/thrift.py
ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c
src/main/python/apache/aurora/executor/aurora_executor.py
636b23d30a897b557eb8c3f8733c90b23cb807ef
src/main/python/apache/aurora/executor/bin/thermos_executor_main.py
9df9b4b79c0c7d29c5088409bf15c0d32a621df0
src/main/python/apache/aurora/executor/common/sandbox.py
f47a32b3fefb4a89940b1ddc473b8316ac00df12
src/main/python/apache/aurora/executor/thermos_task_runner.py
5e4bd65537d186459003c0b9434f1b769e04f448
src/main/python/apache/thermos/config/schema_base.py
f9143cc1b83143d6147f59d90c79435d055d0518
Re: Review Request 29491: [AURORA-986] RequestLoggerTest failed for Locale.FRANCE
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29491/ --- (Updated jan. 5, 2015, 9:34 après-midi) Review request for Aurora, Kevin Sweeney and Bill Farner. Changes --- Incorporate review feedback Repository: aurora Description --- [AURORA-986] RequestLoggerTest failed for Locale.FRANCE Diffs (updated) - src/test/java/org/apache/aurora/scheduler/http/RequestLoggerTest.java 298a34e1a23716cf63ccc02c0ebc9945490fd7bb Diff: https://reviews.apache.org/r/29491/diff/ Testing --- All tests in RequestLoggerTest pass now. Thanks, Issam EL ATIF
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 5, 2015, 11:53 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ead9d28100673440168a32d114ecaa15874978a6 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 5, 2015, 7:56 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Changes --- Updated docs and refactored CommandUtil slightly. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample script is in examples/jobs/docker, as well as an example aurora file. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. The thermos root path defaults to /var/run/thermos, however if a different path is used, it must be passed to the scheduler via `--thermos_observer_root=some path` Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ead9d28100673440168a32d114ecaa15874978a6 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 5, 2015, 8:25 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description (updated) --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ead9d28100673440168a32d114ecaa15874978a6 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29216: Removing client v1 code.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29216/#review66715 --- Ship it! These changes LTGM. I'm not a fan of the churn in the admin code but it seems unavoidable. - Zameer Manji On Dec. 22, 2014, 5:35 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29216/ --- (Updated Dec. 22, 2014, 5:35 p.m.) Review request for Aurora, Kevin Sweeney and Zameer Manji. Bugs: AURORA-775 https://issues.apache.org/jira/browse/AURORA-775 Repository: aurora Description --- Despite the diff size, the changes are very limited in scope. The majority of green/yellow is consolidating admin-related code under /apache/aurora/admin. The rest is removals and renamings to get rid of v2. This diff has minor overlaps with https://reviews.apache.org/r/28971 in vagrant and target renaming parts. Diffs - examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 src/main/python/apache/aurora/admin/BUILD f874264bdf07a9cbb2f0990739be3c95f851b040 src/main/python/apache/aurora/client/BUILD a2a29ff53ffb0088909a10e1673f9578425df5ce src/main/python/apache/aurora/client/api/BUILD 65e5a85e23c4c698356c8b45c45943e560c1bcd5 src/main/python/apache/aurora/client/api/command_runner.py 48cb567c2098620e0ee322fe9528e167ce7c7c62 src/main/python/apache/aurora/client/api/disambiguator.py 6a78ccd44533ef327f751a08c9e2e16555354d97 src/main/python/apache/aurora/client/base.py 258c1fa87645ec48854198529392c84ae08c93be src/main/python/apache/aurora/client/bin/BUILD 671bd78148788d885e50a93bdc006361297a63a2 src/main/python/apache/aurora/client/bin/__init__.py 0663a9ad8ae194b63ce6d77dabfe65564e2d26dd src/main/python/apache/aurora/client/bin/aurora_admin.py 136cf601f0cc137f5111f1ddec5fd823375626b6 src/main/python/apache/aurora/client/bin/aurora_client.py 4999265af358d06d236e819bd600426f4574742c src/main/python/apache/aurora/client/cli/BUILD e61cdfb5f3370ac1c5069632d4158f5ee641bc3a src/main/python/apache/aurora/client/cli/bridge.py 227abf8e21186dd079bc6ce6ff0408a973ad3ab8 src/main/python/apache/aurora/client/cli/context.py f062afc8de1769f6796fc307107c61a59e9366c5 src/main/python/apache/aurora/client/cli/jobs.py 8bdb7f488f68b10b767ced627e55e57698e4f5f2 src/main/python/apache/aurora/client/cli/update.py d1d0b2946c0513381345d90fabe22c1b7c6b8ca9 src/main/python/apache/aurora/client/commands/BUILD 78a2f57b4b42edf363f40e2988cf9a69c36ad003 src/main/python/apache/aurora/client/commands/__init__.py src/main/python/apache/aurora/client/commands/admin.py b7cbba02860af576d9d09e54641d261d7e32c56e src/main/python/apache/aurora/client/commands/core.py cbaa67a78533fedf679eb993c9deb0f09b705616 src/main/python/apache/aurora/client/commands/help.py f79e9718fa9fd8f69331d60e95988f89105089f4 src/main/python/apache/aurora/client/commands/maintenance.py 5321eb6e8d8a15a5e2115e3664d5f12990b67617 src/main/python/apache/aurora/client/commands/run.py 6844ac3eb977f5debfad003eb9e6590d3bb2022e src/main/python/apache/aurora/client/commands/ssh.py ab6832ccb53335ed044b2708abfdac041253f676 src/main/python/apache/aurora/client/options.py f43aced0131cea3adac018d8abcd938ad45af35f src/test/python/apache/aurora/admin/BUILD 3a216809d1e31247f7d01451fcc7fd877a4c1fb2 src/test/python/apache/aurora/client/BUILD e40c7a6cb0e86d0436aff18341a5fc8f3bf9faf8 src/test/python/apache/aurora/client/api/BUILD 2c0c4070cc1f1784b1d4e7f9cd8aac236e97be75 src/test/python/apache/aurora/client/api/api_util.py src/test/python/apache/aurora/client/api/test_api.py 7390d1c4efc5f7b2ec140590de4a293958f73a87 src/test/python/apache/aurora/client/api/test_disambiguator.py 72e98d266e96104552e6891dad737701a04f29c1 src/test/python/apache/aurora/client/api/test_job_monitor.py 1e7a8793e821da3ed53e0ea2b073ea434993a2c6 src/test/python/apache/aurora/client/api/test_quota_check.py 2fc76d21ca63ae7f33b1e03ccb88f52fe82dc76c src/test/python/apache/aurora/client/api/test_task_util.py 048aff6874259810efea463df1ca2a1fdc419ca1 src/test/python/apache/aurora/client/cli/BUILD 4b5817e667281525295730664f96dedeb3e02779 src/test/python/apache/aurora/client/cli/test_bridge.py 6c40bf74677aa6306657dd30016cc380ac20f989 src/test/python/apache/aurora/client/cli/test_cron.py c748212febf5867f5f7cc54e34bf91a8890d src/test/python/apache/aurora/client/cli/util.py 6dba1855f8951d38882c0d5b01a5a609b774d00f src/test/python/apache/aurora/client/commands/BUILD 86ab65e40536a28bddbf8efc9148111ba1518465
Re: Review Request 28731: Implemented TaskScheduler benchmarks.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28731/#review66714 --- src/jmh/java/org/apache/aurora/benchmark/Hosts.java https://reviews.apache.org/r/28731/#comment110299 It would be nice to supply the minimal attributes we expect - host and rack designations. this will at least cause diversity computation to be realistic. Seems like all you need to accomplish that is a parameter for the number of hosts per rack. src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java https://reviews.apache.org/r/28731/#comment110313 Did you consider using a fake for `StateManager`? I consider its behavior out of scope for this benchmark, and cutting it out might remove the need for a bunch of additional binding glue. The same can be said (to a lesster extent) about `TaskAssigner`. src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java https://reviews.apache.org/r/28731/#comment110307 Is this only to get the tasks out of PENDING? If so, would it make sense to instead initialize the storage with tasks in the state you want? Seems like that would reduce the setup complexity. src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java https://reviews.apache.org/r/28731/#comment110300 `getClusterUtilization`? src/jmh/java/org/apache/aurora/benchmark/Tasks.java https://reviews.apache.org/r/28731/#comment110301 This might be worth omitting, i fear its prominent inclusion here could be misleading about how priority is used in scheduling. src/jmh/java/org/apache/aurora/benchmark/fakes/FakeClusterState.java https://reviews.apache.org/r/28731/#comment110309 Did you consider using `CachedClusterState` instead? That might cause some other things to fall into place, and alleviate the manual maintenance of this mapping. AFAICT there's a small amount of wiring necessary to make this work (including a more real `EventSink`), but that might be a worthwhile investment if it simplifies the benchmark implementations. - Bill Farner On Dec. 15, 2014, 10:15 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28731/ --- (Updated Dec. 15, 2014, 10:15 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- Added baseline benchmarks for a few static veto cases. Diffs - build.gradle f9f71a84493b782e9f6072e44e89a2c017cf2a09 src/jmh/java/org/apache/aurora/benchmark/Hosts.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/Offers.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java 5cecada93e4e04b689e826af49f691ed7e94ae49 src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/Tasks.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeClusterState.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeRescheduleCalculator.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeStatsProvider.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java b6402ae42e3c7e4dca1c120fa6ef82d2d69e69d5 src/main/java/org/apache/aurora/scheduler/async/preemptor/ClusterState.java f7e157c890b5627411acd4bd5c2559ef4829147c src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d Diff: https://reviews.apache.org/r/28731/diff/ Testing --- Sample run on a local box: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 10 iterations, 1 s each # Measurement: 100 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Average time, time/op # Benchmark: org.apache.aurora.benchmark.SchedulingBenchmarks.ConstraintMismatchsSchedulingBenchmark.runBenchmark # Run progress: 0.00% complete, ETA 00:05:30 # Fork: 1 of 1 # Warmup Iteration 1: 278284250.000 ns/op # Warmup Iteration 2: 70294312.500 ns/op # Warmup Iteration 3: 19293379.310 ns/op # Warmup Iteration 4: 11945387.097 ns/op # Warmup Iteration 5: 10725388.350 ns/op # Warmup Iteration 6: 13043282.353 ns/op # Warmup Iteration 7: 9233083.333 ns/op # Warmup Iteration 8: 9521051.724 ns/op # Warmup Iteration 9: 10750854.369 ns/op # Warmup Iteration 10: 7460243.243 ns/op
Re: Review Request 29463: Upgrade pystachio to 0.7.4.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29463/#review66677 --- Ship it! Ship It! - Zameer Manji On Dec. 29, 2014, 11:26 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29463/ --- (Updated Dec. 29, 2014, 11:26 a.m.) Review request for Aurora and Kevin Sweeney. Repository: aurora Description --- Upgrade pystachio to 0.7.4. This fixes a bug with loading a serialized pystachio config with multiply nested include statements. Diffs - 3rdparty/python/requirements.txt 05bbf338196cced9f01bf1fadf65682e14dbdf99 Diff: https://reviews.apache.org/r/29463/diff/ Testing --- build-support/jenkins/build.sh Thanks, Brian Wickman
Re: Review Request 29491: [AURORA-986] RequestLoggerTest failed for Locale.FRANCE
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29491/#review66679 --- Could you add some reviewers to this patch? I think adding maxim and kevints should be sufficent. - Zameer Manji On Dec. 30, 2014, 1:04 p.m., Issam EL ATIF wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29491/ --- (Updated Dec. 30, 2014, 1:04 p.m.) Review request for Aurora. Repository: aurora Description --- [AURORA-986] RequestLoggerTest failed for Locale.FRANCE Diffs - src/test/java/org/apache/aurora/scheduler/http/RequestLoggerTest.java 298a34e1a23716cf63ccc02c0ebc9945490fd7bb Diff: https://reviews.apache.org/r/29491/diff/ Testing --- All tests in RequestLoggerTest pass now. Thanks, Issam EL ATIF
Re: Review Request 29491: [AURORA-986] RequestLoggerTest failed for Locale.FRANCE
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29491/ --- (Updated Jan. 5, 2015, 6:36 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Changes --- Adding reviewers -wfarner Repository: aurora Description --- [AURORA-986] RequestLoggerTest failed for Locale.FRANCE Diffs - src/test/java/org/apache/aurora/scheduler/http/RequestLoggerTest.java 298a34e1a23716cf63ccc02c0ebc9945490fd7bb Diff: https://reviews.apache.org/r/29491/diff/ Testing --- All tests in RequestLoggerTest pass now. Thanks, Issam EL ATIF
Re: Review Request 29491: [AURORA-986] RequestLoggerTest failed for Locale.FRANCE
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29491/#review66680 --- Ship it! LGTM with a few nits. src/test/java/org/apache/aurora/scheduler/http/RequestLoggerTest.java https://reviews.apache.org/r/29491/#comment110262 move declaration to the line before the value is used src/test/java/org/apache/aurora/scheduler/http/RequestLoggerTest.java https://reviews.apache.org/r/29491/#comment110263 ditto - Bill Farner On Jan. 5, 2015, 6:36 p.m., Issam EL ATIF wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29491/ --- (Updated Jan. 5, 2015, 6:36 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- [AURORA-986] RequestLoggerTest failed for Locale.FRANCE Diffs - src/test/java/org/apache/aurora/scheduler/http/RequestLoggerTest.java 298a34e1a23716cf63ccc02c0ebc9945490fd7bb Diff: https://reviews.apache.org/r/29491/diff/ Testing --- All tests in RequestLoggerTest pass now. Thanks, Issam EL ATIF
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Jan. 5, 2015, 6:47 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Changes --- Adding reviewers -wfarner Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 src/test/python/apache/aurora/executor/common/test_announcer.py e329a90b8fba43611f5120e2a5ee82220dbe2a91 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Review Request 29586: Replace twitter.common.python dependency with a direct pex dependency (at the latest version).
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/ --- Review request for Aurora, Kevin Sweeney and Brian Wickman. Repository: aurora Description --- Replace twitter.common.python dependency with a direct pex dependency (at the latest version). Diffs - 3rdparty/python/requirements.txt 05bbf338196cced9f01bf1fadf65682e14dbdf99 src/main/python/apache/aurora/client/cli/BUILD e61cdfb5f3370ac1c5069632d4158f5ee641bc3a src/main/python/apache/aurora/client/commands/BUILD 78a2f57b4b42edf363f40e2988cf9a69c36ad003 src/main/python/apache/aurora/common/BUILD 1c6464d8a91a84ca74191814edacaac5e83b78e8 src/main/python/apache/aurora/common/pex_version.py 6aecd8a14eff7cd58becbecc8b05ea193a6c9cec src/main/python/apache/aurora/executor/BUILD 72d1ec5e891a4b7b5101ae913f7520609ccd98a8 src/main/python/apache/aurora/executor/executor_vars.py 7c018271724ffab2ff6930e5802a48b50a39dded src/test/python/apache/aurora/common/test_pex_version.py 7280f703463c6205493a718310f20a7fd21a0c6b Diff: https://reviews.apache.org/r/29586/diff/ Testing --- ./pants build src/test/python/apache/aurora:all bash src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Joshua Cohen
Re: Review Request 29586: Replace twitter.common.python dependency with a direct pex dependency (at the latest version).
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/#review66685 --- src/main/python/apache/aurora/client/cli/BUILD https://reviews.apache.org/r/29586/#comment110267 nit: keep these sorted, here and below - Kevin Sweeney On Jan. 5, 2015, 10:52 a.m., Joshua Cohen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/ --- (Updated Jan. 5, 2015, 10:52 a.m.) Review request for Aurora, Kevin Sweeney and Brian Wickman. Repository: aurora Description --- Replace twitter.common.python dependency with a direct pex dependency (at the latest version). Diffs - 3rdparty/python/requirements.txt 05bbf338196cced9f01bf1fadf65682e14dbdf99 src/main/python/apache/aurora/client/cli/BUILD e61cdfb5f3370ac1c5069632d4158f5ee641bc3a src/main/python/apache/aurora/client/commands/BUILD 78a2f57b4b42edf363f40e2988cf9a69c36ad003 src/main/python/apache/aurora/common/BUILD 1c6464d8a91a84ca74191814edacaac5e83b78e8 src/main/python/apache/aurora/common/pex_version.py 6aecd8a14eff7cd58becbecc8b05ea193a6c9cec src/main/python/apache/aurora/executor/BUILD 72d1ec5e891a4b7b5101ae913f7520609ccd98a8 src/main/python/apache/aurora/executor/executor_vars.py 7c018271724ffab2ff6930e5802a48b50a39dded src/test/python/apache/aurora/common/test_pex_version.py 7280f703463c6205493a718310f20a7fd21a0c6b Diff: https://reviews.apache.org/r/29586/diff/ Testing --- ./pants build src/test/python/apache/aurora:all bash src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Joshua Cohen
Re: Review Request 29586: Replace twitter.common.python dependency with a direct pex dependency (at the latest version).
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/#review66687 --- Can you add details about why this is being done? At a quick glance, it appears as though we're relying on transitive dependencies of pex, but i suspect there's more to it. - Bill Farner On Jan. 5, 2015, 6:52 p.m., Joshua Cohen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/ --- (Updated Jan. 5, 2015, 6:52 p.m.) Review request for Aurora, Kevin Sweeney and Brian Wickman. Repository: aurora Description --- Replace twitter.common.python dependency with a direct pex dependency (at the latest version). Diffs - 3rdparty/python/requirements.txt 05bbf338196cced9f01bf1fadf65682e14dbdf99 src/main/python/apache/aurora/client/cli/BUILD e61cdfb5f3370ac1c5069632d4158f5ee641bc3a src/main/python/apache/aurora/client/commands/BUILD 78a2f57b4b42edf363f40e2988cf9a69c36ad003 src/main/python/apache/aurora/common/BUILD 1c6464d8a91a84ca74191814edacaac5e83b78e8 src/main/python/apache/aurora/common/pex_version.py 6aecd8a14eff7cd58becbecc8b05ea193a6c9cec src/main/python/apache/aurora/executor/BUILD 72d1ec5e891a4b7b5101ae913f7520609ccd98a8 src/main/python/apache/aurora/executor/executor_vars.py 7c018271724ffab2ff6930e5802a48b50a39dded src/test/python/apache/aurora/common/test_pex_version.py 7280f703463c6205493a718310f20a7fd21a0c6b Diff: https://reviews.apache.org/r/29586/diff/ Testing --- ./pants build src/test/python/apache/aurora:all bash src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Joshua Cohen
Re: Review Request 29472: [WIP] Updates documentation markdown to a consistent style.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29472/#review66690 --- Ship it! Thank you! I've been doing some drive-by fixes like this, thanks for the broad cleanup! - Bill Farner On Dec. 30, 2014, 10:18 p.m., Dave Lester wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29472/ --- (Updated Dec. 30, 2014, 10:18 p.m.) Review request for Aurora. Bugs: AURORA-983 https://issues.apache.org/jira/browse/AURORA-983 Repository: aurora Description --- Updates documentation markdown to a consistent style. WIP. Still needs a style guide typed up to formalize the style of markdown. Diffs - docs/client-commands.md 6c929c1bae058d8dfcac3bb4220d73d8e389a568 docs/clientv2.md 471d80cae8f05a52f50a1aa4635209354487863d docs/committers.md 23a25c1e78450690102120ed61afb3f37b69c9c2 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/configuration-tutorial.md b64dc14649939083821ac682b08a5d3eb6e8ae41 docs/contributing.md 0ac89543a75921397a80b0473b52bbeea0ed9a30 docs/cron-jobs.md c3e8eede307fa94807bd65b34d9c24711c8269c1 docs/developing-aurora-client.md b9912bce44d65ddd7f1e35f0ea9356a89d5fe767 docs/developing-aurora-scheduler.md 7f6cc2e6c8e01115a9b7a7dc7633bcd88ba02a0f docs/hooks.md 533c81df9a5934ea903e3dbfb9fca6a211ceba21 docs/monitoring.md 8aee66915f40d39368b57acd2b9e328e52b8cdbe docs/resource-isolation.md 7e8d88d09093d85c07c84bd3d6476fc89ff21c3b docs/scheduler-storage.md 1cd02f87d3062d76942cb9ec755084adc64d0aec docs/sla.md 14e9108fda91200bbf56384c96b9cd926689311f docs/storage-config.md 971bc1673f8be800f0d081a44018fc17a5ed025a docs/storage.md 6ffed541d1579a8131ce642b082be602431329d3 docs/test-resource-generation.md 335586d64757f1e6293a89f14c1c3d578321eac6 docs/tutorial.md e46addd664ae6219b10b6a57822efeabf16ede40 docs/user-guide.md 877f5b7f44e58dd102a6111635e1b669ef24508f docs/vagrant.md c3632eebfe4c65d7dec6fd6cd02b45308691914d Diff: https://reviews.apache.org/r/29472/diff/ Testing --- Thanks, Dave Lester
Re: Review Request 29586: Replace twitter.common.python dependency with a direct pex dependency (at the latest version).
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/ --- (Updated Jan. 5, 2015, 7:07 p.m.) Review request for Aurora, Kevin Sweeney and Brian Wickman. Changes --- Sort BUILD dependencies. Repository: aurora Description --- Replace twitter.common.python dependency with a direct pex dependency (at the latest version). Diffs (updated) - 3rdparty/python/requirements.txt 05bbf338196cced9f01bf1fadf65682e14dbdf99 src/main/python/apache/aurora/client/cli/BUILD e61cdfb5f3370ac1c5069632d4158f5ee641bc3a src/main/python/apache/aurora/client/commands/BUILD 78a2f57b4b42edf363f40e2988cf9a69c36ad003 src/main/python/apache/aurora/common/BUILD 1c6464d8a91a84ca74191814edacaac5e83b78e8 src/main/python/apache/aurora/common/pex_version.py 6aecd8a14eff7cd58becbecc8b05ea193a6c9cec src/main/python/apache/aurora/executor/BUILD 72d1ec5e891a4b7b5101ae913f7520609ccd98a8 src/main/python/apache/aurora/executor/executor_vars.py 7c018271724ffab2ff6930e5802a48b50a39dded src/test/python/apache/aurora/common/test_pex_version.py 7280f703463c6205493a718310f20a7fd21a0c6b Diff: https://reviews.apache.org/r/29586/diff/ Testing --- ./pants build src/test/python/apache/aurora:all bash src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Joshua Cohen
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review66692 --- src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java https://reviews.apache.org/r/28920/#comment110272 This can't be made private due to the way @Timed works (for more information see https://github.com/google/guice/wiki/AOP). - Kevin Sweeney On Dec. 26, 2014, 1:05 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Dec. 26, 2014, 1:05 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample script is in examples/jobs/docker, as well as an example aurora file. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. The thermos root path defaults to /var/run/thermos, however if a different path is used, it must be passed to the scheduler via `--thermos_observer_root=some path` Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ead9d28100673440168a32d114ecaa15874978a6 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29586: Replace twitter.common.python dependency with a direct pex dependency (at the latest version).
On Jan. 5, 2015, 6:53 p.m., Kevin Sweeney wrote: src/main/python/apache/aurora/client/cli/BUILD, line 36 https://reviews.apache.org/r/29586/diff/1/?file=806793#file806793line36 nit: keep these sorted, here and below Done for the rest (this particular one was in the right place afaict). - Joshua --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/#review66685 --- On Jan. 5, 2015, 7:07 p.m., Joshua Cohen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/ --- (Updated Jan. 5, 2015, 7:07 p.m.) Review request for Aurora, Kevin Sweeney and Brian Wickman. Repository: aurora Description --- Replace twitter.common.python dependency with a direct pex dependency (at the latest version). Diffs - 3rdparty/python/requirements.txt 05bbf338196cced9f01bf1fadf65682e14dbdf99 src/main/python/apache/aurora/client/cli/BUILD e61cdfb5f3370ac1c5069632d4158f5ee641bc3a src/main/python/apache/aurora/client/commands/BUILD 78a2f57b4b42edf363f40e2988cf9a69c36ad003 src/main/python/apache/aurora/common/BUILD 1c6464d8a91a84ca74191814edacaac5e83b78e8 src/main/python/apache/aurora/common/pex_version.py 6aecd8a14eff7cd58becbecc8b05ea193a6c9cec src/main/python/apache/aurora/executor/BUILD 72d1ec5e891a4b7b5101ae913f7520609ccd98a8 src/main/python/apache/aurora/executor/executor_vars.py 7c018271724ffab2ff6930e5802a48b50a39dded src/test/python/apache/aurora/common/test_pex_version.py 7280f703463c6205493a718310f20a7fd21a0c6b Diff: https://reviews.apache.org/r/29586/diff/ Testing --- ./pants build src/test/python/apache/aurora:all bash src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Joshua Cohen
Re: Review Request 29586: Replace twitter.common.python dependency with a direct pex dependency (at the latest version).
On Jan. 5, 2015, 6:55 p.m., Bill Farner wrote: Can you add details about why this is being done? At a quick glance, it appears as though we're relying on transitive dependencies of pex, but i suspect there's more to it. We were depending on all of twitter.common.python for only pex. Depending on pex directly lets us more easily consume upstream pex changes without going through the dance of getting a new version of t.c.p published. - Joshua --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/#review66687 --- On Jan. 5, 2015, 7:07 p.m., Joshua Cohen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/ --- (Updated Jan. 5, 2015, 7:07 p.m.) Review request for Aurora, Kevin Sweeney and Brian Wickman. Repository: aurora Description --- Replace twitter.common.python dependency with a direct pex dependency (at the latest version). Diffs - 3rdparty/python/requirements.txt 05bbf338196cced9f01bf1fadf65682e14dbdf99 src/main/python/apache/aurora/client/cli/BUILD e61cdfb5f3370ac1c5069632d4158f5ee641bc3a src/main/python/apache/aurora/client/commands/BUILD 78a2f57b4b42edf363f40e2988cf9a69c36ad003 src/main/python/apache/aurora/common/BUILD 1c6464d8a91a84ca74191814edacaac5e83b78e8 src/main/python/apache/aurora/common/pex_version.py 6aecd8a14eff7cd58becbecc8b05ea193a6c9cec src/main/python/apache/aurora/executor/BUILD 72d1ec5e891a4b7b5101ae913f7520609ccd98a8 src/main/python/apache/aurora/executor/executor_vars.py 7c018271724ffab2ff6930e5802a48b50a39dded src/test/python/apache/aurora/common/test_pex_version.py 7280f703463c6205493a718310f20a7fd21a0c6b Diff: https://reviews.apache.org/r/29586/diff/ Testing --- ./pants build src/test/python/apache/aurora:all bash src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Joshua Cohen
Re: Review Request 29586: Replace twitter.common.python dependency with a direct pex dependency (at the latest version).
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/#review66697 --- Ship it! Ship It! - Bill Farner On Jan. 5, 2015, 7:07 p.m., Joshua Cohen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/ --- (Updated Jan. 5, 2015, 7:07 p.m.) Review request for Aurora, Kevin Sweeney and Brian Wickman. Repository: aurora Description --- Replace twitter.common.python dependency with a direct pex dependency (at the latest version). Diffs - 3rdparty/python/requirements.txt 05bbf338196cced9f01bf1fadf65682e14dbdf99 src/main/python/apache/aurora/client/cli/BUILD e61cdfb5f3370ac1c5069632d4158f5ee641bc3a src/main/python/apache/aurora/client/commands/BUILD 78a2f57b4b42edf363f40e2988cf9a69c36ad003 src/main/python/apache/aurora/common/BUILD 1c6464d8a91a84ca74191814edacaac5e83b78e8 src/main/python/apache/aurora/common/pex_version.py 6aecd8a14eff7cd58becbecc8b05ea193a6c9cec src/main/python/apache/aurora/executor/BUILD 72d1ec5e891a4b7b5101ae913f7520609ccd98a8 src/main/python/apache/aurora/executor/executor_vars.py 7c018271724ffab2ff6930e5802a48b50a39dded src/test/python/apache/aurora/common/test_pex_version.py 7280f703463c6205493a718310f20a7fd21a0c6b Diff: https://reviews.apache.org/r/29586/diff/ Testing --- ./pants build src/test/python/apache/aurora:all bash src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Joshua Cohen
Re: Review Request 29586: Replace twitter.common.python dependency with a direct pex dependency (at the latest version).
On Jan. 5, 2015, 6:55 p.m., Bill Farner wrote: Can you add details about why this is being done? At a quick glance, it appears as though we're relying on transitive dependencies of pex, but i suspect there's more to it. Joshua Cohen wrote: We were depending on all of twitter.common.python for only pex. Depending on pex directly lets us more easily consume upstream pex changes without going through the dance of getting a new version of t.c.p published. pex is twitter.common.python. they were merged at 0.7.0. versions after that are published as 'pex'. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/#review66687 --- On Jan. 5, 2015, 7:07 p.m., Joshua Cohen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29586/ --- (Updated Jan. 5, 2015, 7:07 p.m.) Review request for Aurora, Kevin Sweeney and Brian Wickman. Repository: aurora Description --- Replace twitter.common.python dependency with a direct pex dependency (at the latest version). Diffs - 3rdparty/python/requirements.txt 05bbf338196cced9f01bf1fadf65682e14dbdf99 src/main/python/apache/aurora/client/cli/BUILD e61cdfb5f3370ac1c5069632d4158f5ee641bc3a src/main/python/apache/aurora/client/commands/BUILD 78a2f57b4b42edf363f40e2988cf9a69c36ad003 src/main/python/apache/aurora/common/BUILD 1c6464d8a91a84ca74191814edacaac5e83b78e8 src/main/python/apache/aurora/common/pex_version.py 6aecd8a14eff7cd58becbecc8b05ea193a6c9cec src/main/python/apache/aurora/executor/BUILD 72d1ec5e891a4b7b5101ae913f7520609ccd98a8 src/main/python/apache/aurora/executor/executor_vars.py 7c018271724ffab2ff6930e5802a48b50a39dded src/test/python/apache/aurora/common/test_pex_version.py 7280f703463c6205493a718310f20a7fd21a0c6b Diff: https://reviews.apache.org/r/29586/diff/ Testing --- ./pants build src/test/python/apache/aurora:all bash src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Joshua Cohen
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 5, 2015, 7:08 p.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java, line 205 https://reviews.apache.org/r/28920/diff/4/?file=801608#file801608line205 This can't be made private due to the way @Timed works (for more information see https://github.com/google/guice/wiki/AOP). Bill Farner wrote: Presumably a rebase will cause this to be caught by Maxim's new PMD rule. I think this might have been an artifact of a rebase conflict, I dont know why I would have changed this myself. Good catch though. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review66692 --- On Dec. 26, 2014, 9:05 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Dec. 26, 2014, 9:05 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample script is in examples/jobs/docker, as well as an example aurora file. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. The thermos root path defaults to /var/run/thermos, however if a different path is used, it must be passed to the scheduler via `--thermos_observer_root=some path` Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ead9d28100673440168a32d114ecaa15874978a6 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/#review66702
---
This is 99% just style nits. Unfortunately our Java styleguide isn't published,
but I'm working on rectifying that!
api/src/main/thrift/org/apache/aurora/gen/api.thrift
https://reviews.apache.org/r/28920/#comment110279
nit: fix indentation, should be 2 spaces, not 4 (same goes for the Mode
enum below).
examples/jobs/docker/hello_docker.aurora
https://reviews.apache.org/r/28920/#comment110282
indent 2 to be consistent w/ the task below?
examples/jobs/docker/hello_docker.aurora
https://reviews.apache.org/r/28920/#comment110280
put this on a single line?
examples/jobs/docker/hello_docker.aurora
https://reviews.apache.org/r/28920/#comment110281
move to previous line.
examples/vagrant/aurorabuild.sh
https://reviews.apache.org/r/28920/#comment110283
Fix indentation, avoid tabs
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java
https://reviews.apache.org/r/28920/#comment110284
Keep indentation consistent with the other args below (indent `help =
...` four spaces).
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java
https://reviews.apache.org/r/28920/#comment110286
Insteat of concatenating the strings together just put the help on a
separate line (applies to all instances of this style below).
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java
https://reviews.apache.org/r/28920/#comment110288
Do we need to plumb the wrapper path all the way through to the
CommandUtil? What if instead of passing both along we figure out the correct
path to use here at start up and passed that along directly?
I.e. we could do something like...
OptionalString executorPath =
Optional.of(THERMOS_EXECUTOR_PATH.get()).or(Optional.of(THERMOS_EXECUTOR_WRAPPER_PATH.get()));
if (!executorPath.isPresent()) {
throw new IllegalStateException(...);
}
bind(ExecutorSettings.class).toInstance(new ExecutorSettings(
executorPath.get(),
...));
src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java
https://reviews.apache.org/r/28920/#comment110302
style nit: method continuation should be formatted like:
public static void create(
String executorUri,
String wrapperUri,
String basePath,
CommandInfo.Builder builder) {
String uriToAdd;
...
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
https://reviews.apache.org/r/28920/#comment110303
This could be inlined into the addVolumes call
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
https://reviews.apache.org/r/28920/#comment110304
Fits on one line (we use 100 chars as the wrap point).
src/main/python/apache/thermos/config/schema_base.py
https://reviews.apache.org/r/28920/#comment110305
2 blank lines between top level constructs (c.f.
https://www.python.org/dev/peps/pep-0008/#blank-lines).
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
https://reviews.apache.org/r/28920/#comment110306
Fix indentation, should be 2 chars.
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
https://reviews.apache.org/r/28920/#comment110308
one param per line when exceeding line length:
new ExecutorSettings(
EXECUTOR_PATH,
...);
src/test/python/apache/aurora/executor/test_thermos_executor.py
https://reviews.apache.org/r/28920/#comment110310
This seems like a change in semantics... the sandbox_provider previously
was expected to be a factory function that returned the sandbox, now you're
passing in the sandbox itself? Why the change?
- Joshua Cohen
On Jan. 5, 2015, 8:25 p.m., Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/
---
(Updated Jan. 5, 2015, 8:25 p.m.)
Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner.
Bugs: AURORA-633
https://issues.apache.org/jira/browse/AURORA-633
Repository: aurora
Description
---
This change adds support for launching docker containers through aurora.
These changes are based off of the discussion in
https://issues.apache.org/jira/browse/AURORA-633
As of now, a special thermos_executor.sh script is needed to launch the
executor inside docker containers. A sample aurora file is in
examples/jobs/docker.
In addition, mesos-slave must be run with `--containerizers=docker,mesos`,
the example upstart config in
