Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69420 --- Master (3fa004b) is red with this patch. ./build-support/jenkins/build.sh src.test.python.apache.aurora.client.cli.config . SUCCESS src.test.python.apache.aurora.client.cli.cron . SUCCESS src.test.python.apache.aurora.client.cli.inspect . SUCCESS src.test.python.apache.aurora.client.cli.job . SUCCESS src.test.python.apache.aurora.client.cli.plugins . SUCCESS src.test.python.apache.aurora.client.cli.quota . SUCCESS src.test.python.apache.aurora.client.cli.sla . SUCCESS src.test.python.apache.aurora.client.cli.supdate . SUCCESS src.test.python.apache.aurora.client.cli.task . SUCCESS src.test.python.apache.aurora.client.cli.update . SUCCESS src.test.python.apache.aurora.client.cli.version . SUCCESS src.test.python.apache.aurora.client.config . SUCCESS src.test.python.apache.aurora.client.hooks.hooked_api . SUCCESS src.test.python.apache.aurora.client.hooks.non_hooked_api . SUCCESS src.test.python.apache.aurora.common.test_aurora_job_key . SUCCESS src.test.python.apache.aurora.common.test_cluster . SUCCESS src.test.python.apache.aurora.common.test_cluster_option . SUCCESS src.test.python.apache.aurora.common.test_clusters . SUCCESS src.test.python.apache.aurora.common.test_http_signaler . SUCCESS src.test.python.apache.aurora.common.test_pex_version . SUCCESS src.test.python.apache.aurora.common.test_shellify . SUCCESS src.test.python.apache.aurora.common.test_transport . SUCCESS src.test.python.apache.aurora.config.test_base . SUCCESS src.test.python.apache.aurora.config.test_constraint_parsing . SUCCESS src.test.python.apache.aurora.config.test_loader . SUCCESS src.test.python.apache.aurora.config.test_thrift . SUCCESS src.test.python.apache.aurora.executor.common.task_info . SUCCESS src.test.python.apache.aurora.executor.executor_base . SUCCESS src.test.python.apache.aurora.executor.executor_detector . SUCCESS src.test.python.apache.aurora.executor.executor_vars . SUCCESS src.test.python.apache.aurora.executor.status_manager . SUCCESS src.test.python.apache.aurora.executor.thermos_task_runner . FAILURE src.test.python.apache.thermos.common.test_pathspec . SUCCESS src.test.python.apache.thermos.core.test_runner_integration . SUCCESS src.test.python.apache.thermos.monitoring.test_disk . SUCCESS FAILURE [31m FAILURE[0m I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 23, 2015, 5:23 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 23, 2015, 5:23 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69404 --- Master (3fa004b) is red with this patch. ./build-support/jenkins/build.sh SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/cli/test_inspect.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/cli/test_diff.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/cli/test_cancel_update.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/cli/__init__.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/cli/test_status.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/cli/test_plugins.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/cli/test_update.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/cli/test_task.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/cli/util.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/cli/test_create.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/test_task_util.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/test_job_monitor.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/test_restarter.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/test_health_check.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/test_instance_watcher.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/test_scheduler_mux.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/test_sla.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/test_api.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/test_updater_util.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/__init__.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/test_updater.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/test_quota_check.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/test_scheduler_client.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/hooks/test_hooked_api.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/hooks/test_non_hooked_api.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/thermos/core/test_staged_kill.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/thermos/core/test_helper.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/thermos/core/test_angry.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/thermos/core/test_process.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/thermos/core/test_runner_integration.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/thermos/core/test_finalization.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/thermos/core/test_failure_limit.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/thermos/core/test_ephemerals.py Everything Looks Good! SUCCESS:
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69439 --- @ReviewBot retry - Steve Niemitz On Jan. 23, 2015, 5:23 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 23, 2015, 5:23 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh f692f025a7e5f2f0dddb7f6c81ea12fcb8272020 examples/vagrant/provision-dev-cluster.sh 40c165925c2110fb727c66ff5a34cf5ab8415343 examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 8428941c10641857a952f34df4e46a8fac5476a8 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py f29faf1d51baa4af66ad8c6579ffa354409e9536 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 41200bd5d74ee3239279567d79a5f48fb6af2e44 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 9fd188fb8f004d0a7664420bfda56568cebedb6f src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 7eafe074b686d55ad96018006cf4acfa823513c3 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java ad9126c32893080e128d086ea3bfd7ad23d27b89 src/test/python/apache/aurora/client/cli/test_status.py e531fa06e508d9792af51c62e67120c21baa7a81 src/test/python/apache/aurora/executor/test_thermos_executor.py 6c8ae1cf7fb578237708a954bd42162d66b39f4d Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 30224: Stream backups directly to file.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30224/#review69443 --- Ship it! - Bill Farner On Jan. 23, 2015, 7:19 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30224/ --- (Updated Jan. 23, 2015, 7:19 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1053 https://issues.apache.org/jira/browse/AURORA-1053 Repository: aurora Description --- Stream backups directly to file. Diffs - src/main/java/org/apache/aurora/scheduler/storage/backup/StorageBackup.java 69521d395d6f8df17c3fbcdb12633a06c62e2136 Diff: https://reviews.apache.org/r/30224/diff/ Testing --- ./gradlew build Thanks, Kevin Sweeney
Re: Review Request 30224: Stream backups directly to file.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30224/#review69444 --- Ship it! Ship It! - Zameer Manji On Jan. 23, 2015, 11:19 a.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30224/ --- (Updated Jan. 23, 2015, 11:19 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1053 https://issues.apache.org/jira/browse/AURORA-1053 Repository: aurora Description --- Stream backups directly to file. Diffs - src/main/java/org/apache/aurora/scheduler/storage/backup/StorageBackup.java 69521d395d6f8df17c3fbcdb12633a06c62e2136 Diff: https://reviews.apache.org/r/30224/diff/ Testing --- ./gradlew build Thanks, Kevin Sweeney
Re: Review Request 30224: Stream backups directly to file.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30224/#review69445 --- Ship it! Ship It! - Zameer Manji On Jan. 23, 2015, 11:25 a.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30224/ --- (Updated Jan. 23, 2015, 11:25 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1053 https://issues.apache.org/jira/browse/AURORA-1053 Repository: aurora Description --- Stream backups directly to file. Diffs - src/main/java/org/apache/aurora/scheduler/storage/backup/StorageBackup.java 69521d395d6f8df17c3fbcdb12633a06c62e2136 Diff: https://reviews.apache.org/r/30224/diff/ Testing --- ./gradlew build Thanks, Kevin Sweeney
Review Request 30224: Stream backups directly to file.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30224/ --- Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1053 https://issues.apache.org/jira/browse/AURORA-1053 Repository: aurora Description --- Stream backups directly to file. Diffs - src/main/java/org/apache/aurora/scheduler/storage/backup/StorageBackup.java 69521d395d6f8df17c3fbcdb12633a06c62e2136 Diff: https://reviews.apache.org/r/30224/diff/ Testing --- ./gradlew build Thanks, Kevin Sweeney
Review Request 30225: Modifying update controller to support heartbeats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30225/ --- Review request for Aurora, David McLaughlin, Joshua Cohen, and Bill Farner. Bugs: AURORA-1010 https://issues.apache.org/jira/browse/AURORA-1010 Repository: aurora Description --- Added pulsing support into the JobUpdateController. The qualified coordinated updates get blocked until a pulse arrives. An update then becomes active and proceeds until `blockIfNoPulsesAfterMs` expires or the update reaches a terminal state (whichever comes first). Not particularly happy with plumbing through OneWayJobUpdater but the alternative is a state machine change, which is much hairier and will require more changes in the JobUpdaterController. Going with the minimal diff here. Diffs - src/main/java/org/apache/aurora/scheduler/updater/JobUpdateController.java d3b30d48b76d8d7c64cda006a34f7ed3296526f2 src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java a992938d4e12b20f81608be6bbdc24c0a211c3fd src/main/java/org/apache/aurora/scheduler/updater/OneWayJobUpdater.java 27a5b9026f5ac3b3bdeb32813b10435bc3dab173 src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 4c827b183a87b4d97774edbfaa960bd1c3de72a5 src/test/java/org/apache/aurora/scheduler/updater/OneWayJobUpdaterTest.java 7d0a7438b4a517e5e0d44f4e99aceb1a6d19f987 Diff: https://reviews.apache.org/r/30225/diff/ Testing --- ./gradlew -Pq build Thanks, Maxim Khutornenko
Re: Review Request 30224: Stream backups directly to file.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30224/#review69452 --- Master (3fa004b) is green with this patch. ./build-support/jenkins/build.sh However, it appears that it might lack test coverage. I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 23, 2015, 7:25 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30224/ --- (Updated Jan. 23, 2015, 7:25 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1053 https://issues.apache.org/jira/browse/AURORA-1053 Repository: aurora Description --- Stream backups directly to file. Diffs - src/main/java/org/apache/aurora/scheduler/storage/backup/StorageBackup.java 69521d395d6f8df17c3fbcdb12633a06c62e2136 Diff: https://reviews.apache.org/r/30224/diff/ Testing --- ./gradlew build Thanks, Kevin Sweeney
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69447 --- Ship it! Master (3fa004b) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 23, 2015, 5:23 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 23, 2015, 5:23 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh f692f025a7e5f2f0dddb7f6c81ea12fcb8272020 examples/vagrant/provision-dev-cluster.sh 40c165925c2110fb727c66ff5a34cf5ab8415343 examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 8428941c10641857a952f34df4e46a8fac5476a8 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py f29faf1d51baa4af66ad8c6579ffa354409e9536 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 41200bd5d74ee3239279567d79a5f48fb6af2e44 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 9fd188fb8f004d0a7664420bfda56568cebedb6f src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 7eafe074b686d55ad96018006cf4acfa823513c3 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java ad9126c32893080e128d086ea3bfd7ad23d27b89 src/test/python/apache/aurora/client/cli/test_status.py e531fa06e508d9792af51c62e67120c21baa7a81 src/test/python/apache/aurora/executor/test_thermos_executor.py 6c8ae1cf7fb578237708a954bd42162d66b39f4d Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 30207: Simplify AuroraCommandContext
On Jan. 23, 2015, 12:47 p.m., Maxim Khutornenko wrote: I am not convinced there is enough value in this diff to risk possible regression. Besides, the majority of what this diff touches will die out along with the client updater. Zameer Manji wrote: Is there an ETA for the destruction of the client updater? Maxim Khutornenko wrote: Any time we feel ready to drop beta from scheduler updater. I'm willing to drop this diff, if you are willing to start the conversation on when we can drop 'beta' from the scheduler updater. - Zameer --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/#review69458 --- On Jan. 22, 2015, 7:32 p.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/ --- (Updated Jan. 22, 2015, 7:32 p.m.) Review request for Aurora, Maxim Khutornenko and Bill Farner. Repository: aurora Description --- The AuroraCommandContext class is used in multiple commands and contains common code for all of them. However some portions are only used by one command. This patch takes some of those portions and moves them to the command that requires that functionality. Diffs - src/main/python/apache/aurora/client/cli/context.py 51c7d24dca664e476e62f1864d095416dfab70e4 src/main/python/apache/aurora/client/cli/jobs.py 8f349c09637c16e2499e85f2dc96eb7ccffd0aaf src/main/python/apache/aurora/client/cli/update.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_supdate.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_update.py 8b7d11202b35deb09a248cfe0a96458fb70c Diff: https://reviews.apache.org/r/30207/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/client:: Thanks, Zameer Manji
Re: Review Request 30187: Remove support for cluster metadata in YAML format.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30187/#review69469 --- @ReviewBot retry - Bill Farner On Jan. 22, 2015, 9:09 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30187/ --- (Updated Jan. 22, 2015, 9:09 p.m.) Review request for Aurora, Brian Wickman and Zameer Manji. Bugs: AURORA-1029 https://issues.apache.org/jira/browse/AURORA-1029 Repository: aurora Description --- Remove support for cluster metadata in YAML format. Diffs - src/main/python/apache/aurora/common/clusters.py e55aa774b4b868f696a7de51bb016f950871dd1e src/test/python/apache/aurora/common/BUILD 14165b96be99b8de418f4bb8def9f27eaf29e67d src/test/python/apache/aurora/common/test_clusters.py 45250e609cca1149dc296b2aaf645ff2f58f8288 Diff: https://reviews.apache.org/r/30187/diff/ Testing --- ./build-support/jenkins/build.sh test_end_to_end.sh is currently broken on master, i will address that and ensure it passes before committing this. Thanks, Bill Farner
Re: Review Request 30204: Upgrade to rbt=0.7.0.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30204/#review69473 --- Ship it! Ship It! - Zameer Manji On Jan. 22, 2015, 6:11 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30204/ --- (Updated Jan. 22, 2015, 6:11 p.m.) Review request for Aurora and Kevin Sweeney. Repository: aurora Description --- Release notes: https://www.reviewboard.org/docs/releasenotes/rbtools/0.7/ Some potentially-interesting new features: ``` rbt land The new rbt land command is a quick and easy way to push a change that has been reviewed on Review Board to the upstream repository. The change may be in a local branch or stored as a patch on Review Board. When running against Review Board 2.0+, this command will start by checking the approval state of the review request (on older versions, it just checks for one or more “Ship It!”s). If the change is approved, the patch will be applied just like rbt patch. The change can optionally be pushed to the remote repository using the -p/--push command line option. This is currently only available when using Git repositories. rbt stamp The new rbt stamp command will amend a commit message with a “Reviewed at url” line. This feature is especially useful when using the close-on-submit hooks in Review Board 2.x to close out review requests when the changes are committed. This is currently only supported for Git repositories. Patch by Yanjia (Nicole) Xin. ``` Diffs - rbt 0742cf47924ee013758883878fa229b5b876be27 Diff: https://reviews.apache.org/r/30204/diff/ Testing --- Posted this review. Thanks, Bill Farner
Re: Review Request 30187: Remove support for cluster metadata in YAML format.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30187/#review69478 --- Master (e3b8905) is red with this patch. ./build-support/jenkins/build.sh Running setup.py install for twitter.common.collections Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py (namespace package) Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py (namespace package) Installing /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.collections-0.3.0-py2.7-nspkg.pth Running setup.py install for smmap Running setup.py install for twitter.common.string Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py (namespace package) Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py (namespace package) Installing /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.string-0.3.0-py2.7-nspkg.pth Running setup.py install for twitter.common.options Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py (namespace package) Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py (namespace package) Installing /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.options-0.3.0-py2.7-nspkg.pth Running setup.py install for twitter.common.dirutil Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py (namespace package) Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py (namespace package) Installing /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.dirutil-0.3.0-py2.7-nspkg.pth Running setup.py install for twitter.common.contextutil Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py (namespace package) Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py (namespace package) Installing /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.contextutil-0.3.0-py2.7-nspkg.pth Running setup.py install for twitter.common.lang Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py (namespace package) Skipping installation of /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py (namespace package) Installing /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.lang-0.3.0-py2.7-nspkg.pth Successfully installed twitter.checkstyle pyflakes pep8 GitPython twitter.common.app gitdb twitter.common.process twitter.common.log twitter.common.util twitter.common.collections smmap twitter.common.string twitter.common.options twitter.common.dirutil twitter.common.contextutil twitter.common.lang Cleaning up... T001:ERROR src/main/python/apache/aurora/common/clusters.py:034 Class globals must be UPPER_SNAKE_CASED | name = Required(String) F401:ERROR src/test/python/apache/aurora/common/test_clusters.py:021 'Parser' imported but unused |from apache.aurora.common.clusters import Clusters, Parser I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 22, 2015, 9:09 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30187/
Re: Review Request 30207: Simplify AuroraCommandContext
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/#review69458 --- I am not convinced there is enough value in this diff to risk possible regression. Besides, the majority of what this diff touches will die out along with the client updater. - Maxim Khutornenko On Jan. 23, 2015, 3:32 a.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/ --- (Updated Jan. 23, 2015, 3:32 a.m.) Review request for Aurora, Maxim Khutornenko and Bill Farner. Repository: aurora Description --- The AuroraCommandContext class is used in multiple commands and contains common code for all of them. However some portions are only used by one command. This patch takes some of those portions and moves them to the command that requires that functionality. Diffs - src/main/python/apache/aurora/client/cli/context.py 51c7d24dca664e476e62f1864d095416dfab70e4 src/main/python/apache/aurora/client/cli/jobs.py 8f349c09637c16e2499e85f2dc96eb7ccffd0aaf src/main/python/apache/aurora/client/cli/update.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_supdate.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_update.py 8b7d11202b35deb09a248cfe0a96458fb70c Diff: https://reviews.apache.org/r/30207/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/client:: Thanks, Zameer Manji
Re: Review Request 30225: Modifying update controller to support heartbeats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30225/#review69460 --- Master (3fa004b) is red with this patch. ./build-support/jenkins/build.sh src.test.python.apache.aurora.client.cli.config . SUCCESS src.test.python.apache.aurora.client.cli.cron . SUCCESS src.test.python.apache.aurora.client.cli.inspect . SUCCESS src.test.python.apache.aurora.client.cli.job . SUCCESS src.test.python.apache.aurora.client.cli.plugins . SUCCESS src.test.python.apache.aurora.client.cli.quota . SUCCESS src.test.python.apache.aurora.client.cli.sla . SUCCESS src.test.python.apache.aurora.client.cli.supdate . SUCCESS src.test.python.apache.aurora.client.cli.task . SUCCESS src.test.python.apache.aurora.client.cli.update . SUCCESS src.test.python.apache.aurora.client.cli.version . SUCCESS src.test.python.apache.aurora.client.config . SUCCESS src.test.python.apache.aurora.client.hooks.hooked_api . SUCCESS src.test.python.apache.aurora.client.hooks.non_hooked_api . SUCCESS src.test.python.apache.aurora.common.test_aurora_job_key . SUCCESS src.test.python.apache.aurora.common.test_cluster . SUCCESS src.test.python.apache.aurora.common.test_cluster_option . SUCCESS src.test.python.apache.aurora.common.test_clusters . SUCCESS src.test.python.apache.aurora.common.test_http_signaler . SUCCESS src.test.python.apache.aurora.common.test_pex_version . SUCCESS src.test.python.apache.aurora.common.test_shellify . SUCCESS src.test.python.apache.aurora.common.test_transport . SUCCESS src.test.python.apache.aurora.config.test_base . SUCCESS src.test.python.apache.aurora.config.test_constraint_parsing . SUCCESS src.test.python.apache.aurora.config.test_loader . SUCCESS src.test.python.apache.aurora.config.test_thrift . SUCCESS src.test.python.apache.aurora.executor.common.task_info . SUCCESS src.test.python.apache.aurora.executor.executor_base . SUCCESS src.test.python.apache.aurora.executor.executor_detector . SUCCESS src.test.python.apache.aurora.executor.executor_vars . SUCCESS src.test.python.apache.aurora.executor.status_manager . SUCCESS src.test.python.apache.aurora.executor.thermos_task_runner . FAILURE src.test.python.apache.thermos.common.test_pathspec . SUCCESS src.test.python.apache.thermos.core.test_runner_integration . SUCCESS src.test.python.apache.thermos.monitoring.test_disk . SUCCESS FAILURE [31m FAILURE[0m I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 23, 2015, 8:37 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30225/ --- (Updated Jan. 23, 2015, 8:37 p.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, and Bill
Re: Review Request 30207: Simplify AuroraCommandContext
On Jan. 23, 2015, 8:47 p.m., Maxim Khutornenko wrote: I am not convinced there is enough value in this diff to risk possible regression. Besides, the majority of what this diff touches will die out along with the client updater. Zameer Manji wrote: Is there an ETA for the destruction of the client updater? Maxim Khutornenko wrote: Any time we feel ready to drop beta from scheduler updater. Zameer Manji wrote: I'm willing to drop this diff, if you are willing to start the conversation on when we can drop 'beta' from the scheduler updater. Maxim Khutornenko wrote: How is that related? :) There is nothing pressing us to graduate scheduler updater at this point. There are still bugs to fix and parity features to implement (e.g. heartbeats) before we are ready for prime time. Bill Farner wrote: I share Maxim's general uneasiness about changing behavior in the client-side updater since it is complex and sunsetting. However, i don't share the concern in this diff. The change appears to be very straightforward, especially in `update.py`. Maxim - is there any particular part you're worried about? I just don't see a reason to shuffle things around (no matter how trivial it looks) for a feature that is going away. I view the value of refactoring as making a long term positive impact on readabilty and reusability. This change does not clear the bar for me. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/#review69458 --- On Jan. 23, 2015, 3:32 a.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/ --- (Updated Jan. 23, 2015, 3:32 a.m.) Review request for Aurora, Maxim Khutornenko and Bill Farner. Repository: aurora Description --- The AuroraCommandContext class is used in multiple commands and contains common code for all of them. However some portions are only used by one command. This patch takes some of those portions and moves them to the command that requires that functionality. Diffs - src/main/python/apache/aurora/client/cli/context.py 51c7d24dca664e476e62f1864d095416dfab70e4 src/main/python/apache/aurora/client/cli/jobs.py 8f349c09637c16e2499e85f2dc96eb7ccffd0aaf src/main/python/apache/aurora/client/cli/update.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_supdate.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_update.py 8b7d11202b35deb09a248cfe0a96458fb70c Diff: https://reviews.apache.org/r/30207/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/client:: Thanks, Zameer Manji
Re: Review Request 30207: Simplify AuroraCommandContext
On Jan. 23, 2015, 12:47 p.m., Maxim Khutornenko wrote: I am not convinced there is enough value in this diff to risk possible regression. Besides, the majority of what this diff touches will die out along with the client updater. Is there an ETA for the destruction of the client updater? - Zameer --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/#review69458 --- On Jan. 22, 2015, 7:32 p.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/ --- (Updated Jan. 22, 2015, 7:32 p.m.) Review request for Aurora, Maxim Khutornenko and Bill Farner. Repository: aurora Description --- The AuroraCommandContext class is used in multiple commands and contains common code for all of them. However some portions are only used by one command. This patch takes some of those portions and moves them to the command that requires that functionality. Diffs - src/main/python/apache/aurora/client/cli/context.py 51c7d24dca664e476e62f1864d095416dfab70e4 src/main/python/apache/aurora/client/cli/jobs.py 8f349c09637c16e2499e85f2dc96eb7ccffd0aaf src/main/python/apache/aurora/client/cli/update.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_supdate.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_update.py 8b7d11202b35deb09a248cfe0a96458fb70c Diff: https://reviews.apache.org/r/30207/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/client:: Thanks, Zameer Manji
Re: Review Request 30207: Simplify AuroraCommandContext
On Jan. 23, 2015, 8:47 p.m., Maxim Khutornenko wrote: I am not convinced there is enough value in this diff to risk possible regression. Besides, the majority of what this diff touches will die out along with the client updater. Zameer Manji wrote: Is there an ETA for the destruction of the client updater? Maxim Khutornenko wrote: Any time we feel ready to drop beta from scheduler updater. Zameer Manji wrote: I'm willing to drop this diff, if you are willing to start the conversation on when we can drop 'beta' from the scheduler updater. How is that related? :) There is nothing pressing us to graduate scheduler updater at this point. There are still bugs to fix and parity features to implement (e.g. heartbeats) before we are ready for prime time. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/#review69458 --- On Jan. 23, 2015, 3:32 a.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/ --- (Updated Jan. 23, 2015, 3:32 a.m.) Review request for Aurora, Maxim Khutornenko and Bill Farner. Repository: aurora Description --- The AuroraCommandContext class is used in multiple commands and contains common code for all of them. However some portions are only used by one command. This patch takes some of those portions and moves them to the command that requires that functionality. Diffs - src/main/python/apache/aurora/client/cli/context.py 51c7d24dca664e476e62f1864d095416dfab70e4 src/main/python/apache/aurora/client/cli/jobs.py 8f349c09637c16e2499e85f2dc96eb7ccffd0aaf src/main/python/apache/aurora/client/cli/update.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_supdate.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_update.py 8b7d11202b35deb09a248cfe0a96458fb70c Diff: https://reviews.apache.org/r/30207/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/client:: Thanks, Zameer Manji
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69480 --- this is looking great. just a few small nits and one higher level question re: where the schema should live. src/main/python/apache/aurora/executor/common/sandbox.py https://reviews.apache.org/r/28920/#comment114179 minor nit: snake_case, i.e. _create_symlinks src/main/python/apache/aurora/executor/common/sandbox.py https://reviews.apache.org/r/28920/#comment114182 os.chdir is a surprising side-effect for sandbox.create. could you instead put this in ThermosTaskRunner.start? i.e. where we do self._popen = subprocess.Popen(cmdline_args), Popen takes a cwd= keyword argument, and it seems like the presence of $MESOS_DIRECTORY should be enough signal that we should be setting it, and we don't need to chdir here. src/main/python/apache/aurora/executor/thermos_task_runner.py https://reviews.apache.org/r/28920/#comment114178 minor nit: os.path.join(X, 'sandbox') instead of + '/sandbox' src/main/python/apache/aurora/executor/thermos_task_runner.py https://reviews.apache.org/r/28920/#comment114181 b/c of AURORA-749, I think this can now just be: None if assigned_task.task.container.docker else assigned_task.task.job.role src/main/python/apache/thermos/config/schema_base.py https://reviews.apache.org/r/28920/#comment114180 sort of higher-level question -- since this information is not used by thermos at all, does it make sense in src/main/python/apache/aurora/config/schema/base.py instead? usually Aurora concerns are in Job() and Thermos concerns are in Task(). - Brian Wickman On Jan. 23, 2015, 5:23 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 23, 2015, 5:23 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh f692f025a7e5f2f0dddb7f6c81ea12fcb8272020 examples/vagrant/provision-dev-cluster.sh 40c165925c2110fb727c66ff5a34cf5ab8415343 examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 8428941c10641857a952f34df4e46a8fac5476a8 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py f29faf1d51baa4af66ad8c6579ffa354409e9536 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py
Re: Review Request 30207: Simplify AuroraCommandContext
On Jan. 23, 2015, 8:47 p.m., Maxim Khutornenko wrote: I am not convinced there is enough value in this diff to risk possible regression. Besides, the majority of what this diff touches will die out along with the client updater. Zameer Manji wrote: Is there an ETA for the destruction of the client updater? Any time we feel ready to drop beta from scheduler updater. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/#review69458 --- On Jan. 23, 2015, 3:32 a.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/ --- (Updated Jan. 23, 2015, 3:32 a.m.) Review request for Aurora, Maxim Khutornenko and Bill Farner. Repository: aurora Description --- The AuroraCommandContext class is used in multiple commands and contains common code for all of them. However some portions are only used by one command. This patch takes some of those portions and moves them to the command that requires that functionality. Diffs - src/main/python/apache/aurora/client/cli/context.py 51c7d24dca664e476e62f1864d095416dfab70e4 src/main/python/apache/aurora/client/cli/jobs.py 8f349c09637c16e2499e85f2dc96eb7ccffd0aaf src/main/python/apache/aurora/client/cli/update.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_supdate.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_update.py 8b7d11202b35deb09a248cfe0a96458fb70c Diff: https://reviews.apache.org/r/30207/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/client:: Thanks, Zameer Manji
Re: Review Request 30207: Simplify AuroraCommandContext
On Jan. 23, 2015, 8:47 p.m., Maxim Khutornenko wrote: I am not convinced there is enough value in this diff to risk possible regression. Besides, the majority of what this diff touches will die out along with the client updater. Zameer Manji wrote: Is there an ETA for the destruction of the client updater? Maxim Khutornenko wrote: Any time we feel ready to drop beta from scheduler updater. Zameer Manji wrote: I'm willing to drop this diff, if you are willing to start the conversation on when we can drop 'beta' from the scheduler updater. Maxim Khutornenko wrote: How is that related? :) There is nothing pressing us to graduate scheduler updater at this point. There are still bugs to fix and parity features to implement (e.g. heartbeats) before we are ready for prime time. I share Maxim's general uneasiness about changing behavior in the client-side updater since it is complex and sunsetting. However, i don't share the concern in this diff. The change appears to be very straightforward, especially in `update.py`. Maxim - is there any particular part you're worried about? - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/#review69458 --- On Jan. 23, 2015, 3:32 a.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30207/ --- (Updated Jan. 23, 2015, 3:32 a.m.) Review request for Aurora, Maxim Khutornenko and Bill Farner. Repository: aurora Description --- The AuroraCommandContext class is used in multiple commands and contains common code for all of them. However some portions are only used by one command. This patch takes some of those portions and moves them to the command that requires that functionality. Diffs - src/main/python/apache/aurora/client/cli/context.py 51c7d24dca664e476e62f1864d095416dfab70e4 src/main/python/apache/aurora/client/cli/jobs.py 8f349c09637c16e2499e85f2dc96eb7ccffd0aaf src/main/python/apache/aurora/client/cli/update.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_supdate.py PRE-CREATION src/test/python/apache/aurora/client/cli/test_update.py 8b7d11202b35deb09a248cfe0a96458fb70c Diff: https://reviews.apache.org/r/30207/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/client:: Thanks, Zameer Manji
Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/#review69487 --- Ship it! Ship It! - Zameer Manji On Jan. 23, 2015, 1:48 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 23, 2015, 1:48 p.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
On Jan. 23, 2015, 9:44 p.m., Maxim Khutornenko wrote: src/main/java/org/apache/aurora/scheduler/ResourceSlot.java, line 47 https://reviews.apache.org/r/30203/diff/1/?file=831389#file831389line47 Given peak 170MB, rounding to 256 feels too wasteful. This change predates the bug being fixed here, so i would like to avoid attempting to rehash that. On Jan. 23, 2015, 9:44 p.m., Maxim Khutornenko wrote: src/main/java/org/apache/aurora/scheduler/configuration/Resources.java, line 206 https://reviews.apache.org/r/30203/diff/1/?file=831394#file831394line206 @VisibleForTesting? Done. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/#review69462 --- On Jan. 23, 2015, 2:06 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 23, 2015, 2:06 a.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 23, 2015, 9:48 p.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
On Jan. 23, 2015, 10:14 p.m., Zameer Manji wrote: src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java, line 36 https://reviews.apache.org/r/30203/diff/2/?file=832278#file832278line36 Can you add a ticket to track this TODO? https://issues.apache.org/jira/browse/AURORA-1055 - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/#review69479 --- On Jan. 23, 2015, 9:48 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 23, 2015, 9:48 p.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 24, 2015, 12:33 a.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Changes --- Needed a few updates to the end-to-end tests to get them to pass. This is needed since the very small tasks used were upsized, and 4 instances (used in the job) would not fit in the slave. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_updated.aurora 67d3dbb6ea02baacfbed72e9d70c3109b3a37759 src/test/sh/org/apache/aurora/e2e/validate_serverset.py 66fa965044a4998857b6458582eeb3296558168d Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30010: [AURORA-184] Remove hardcoded 'host' and 'rack' limit constraints
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30010/#review69508 --- src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java https://reviews.apache.org/r/30010/#comment114206 You don't appear to use DISABLE_LEGACY_CONSTRAINTS in the test files so please remove the annotation and make this variable private. src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java https://reviews.apache.org/r/30010/#comment114207 nit: period at the end of the scentence src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java https://reviews.apache.org/r/30010/#comment114208 nit: period at the end of the statement src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java https://reviews.apache.org/r/30010/#comment114209 Can you actually add a test for the false case here? - Zameer Manji On Jan. 23, 2015, 1:20 a.m., Florian Pfeiffer wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30010/ --- (Updated Jan. 23, 2015, 1:20 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-184 https://issues.apache.org/jira/browse/AURORA-184 Repository: aurora Description --- [AURORA-184] Remove hardcoded 'host' and 'rack' limit constraints This is the first step for AURORA-184, that removes the default hostrack limit constraints. The second step that's still missing would be to add s.th. like --default-constraints as start parameter to the scheduler. AURORA-174 could probably be closed with this?(since the rack limit constraint can be configured in the .aurora file) I can't really estimate the effect of my changes in StorageBackfillTestSchedulerThriftInterfaceTest, please have a closer look at the changes I did there. Since this is also my first code submit, comments about codestyleother bad habbits are very appreciated. Diffs - src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 Diff: https://reviews.apache.org/r/30010/diff/ Testing --- Added test for ConfigurationManager.hasName Added test testNoHostAndRackConstraintsAdded, that checks if the constraints are present Tested on vagrant devcluster to see if constraints are also gone in real life Thanks, Florian Pfeiffer
Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/#review69522 --- This patch does not apply cleanly on master (7ba6226), do you need to rebase? I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 24, 2015, 12:33 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 24, 2015, 12:33 a.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_updated.aurora 67d3dbb6ea02baacfbed72e9d70c3109b3a37759 src/test/sh/org/apache/aurora/e2e/validate_serverset.py 66fa965044a4998857b6458582eeb3296558168d Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 28920: Add support for docker containers to aurora
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/#review69505
---
Ship it!
sweet! ship it modulo minor nit below
src/main/python/apache/aurora/executor/thermos_task_runner.py
https://reviews.apache.org/r/28920/#comment114201
os.environ.get('MESOS_DIRECTORY')
(i worry that cwd='.' will actually break something, whereas cwd=None will
be a no-op.)
- Brian Wickman
On Jan. 24, 2015, midnight, Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/
---
(Updated Jan. 24, 2015, midnight)
Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner.
Bugs: AURORA-633
https://issues.apache.org/jira/browse/AURORA-633
Repository: aurora
Description
---
This change adds support for launching docker containers through aurora.
These changes are based off of the discussion in
https://issues.apache.org/jira/browse/AURORA-633
As of now, a special thermos_executor.sh script is needed to launch the
executor inside docker containers. A sample aurora file is in
examples/jobs/docker.
In addition, mesos-slave must be run with `--containerizers=docker,mesos`,
the example upstart config in examples/vagrant/upstart has been updated to
reflect this.
More information is in subsequent review request comments.
Diffs
-
Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc
api/src/main/thrift/org/apache/aurora/gen/api.thrift
08ba1cdf88b712de22c26c04443079282db59ef9
config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5
docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea
docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7
examples/jobs/docker/hello_docker.aurora PRE-CREATION
examples/vagrant/aurorabuild.sh f692f025a7e5f2f0dddb7f6c81ea12fcb8272020
examples/vagrant/provision-dev-cluster.sh
40c165925c2110fb727c66ff5a34cf5ab8415343
examples/vagrant/upstart/aurora-scheduler.conf
788ec254270bca074dae91829c7f4fccdc8f8bb0
examples/vagrant/upstart/mesos-slave.conf
512ce7ecf34042ed68dda55efb2dd0415f8469db
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java
8428941c10641857a952f34df4e46a8fac5476a8
src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java
5226e3d1b303b1773a057078f2911c5ec2aa97f5
src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java
d885b224ec5a1d529347d84e03ba98ab6734a126
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
01b03508afac37b5a8f0ec5c3da1460695e1ef59
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
5cc85f1f87f3b8355c89e8ecac19de1122a079e6
src/main/python/apache/aurora/config/schema/base.py
d7897794c736778983d506c337a1392f3cc0cc20
src/main/python/apache/aurora/config/thrift.py
ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c
src/main/python/apache/aurora/executor/aurora_executor.py
636b23d30a897b557eb8c3f8733c90b23cb807ef
src/main/python/apache/aurora/executor/bin/thermos_executor_main.py
f7d8977e42aa56188799400bf8e12a6886fb4a8f
src/main/python/apache/aurora/executor/common/sandbox.py
f47a32b3fefb4a89940b1ddc473b8316ac00df12
src/main/python/apache/aurora/executor/thermos_task_runner.py
f29faf1d51baa4af66ad8c6579ffa354409e9536
src/main/python/apache/thermos/core/runner.py
41200bd5d74ee3239279567d79a5f48fb6af2e44
src/main/resources/scheduler/assets/configSummary.html
28878908b0c9381e366b71a3135dfc28c542a890
src/main/resources/scheduler/assets/js/services.js
b744f375411e09b7f776e4a05ee5961227143439
src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java
9fd188fb8f004d0a7664420bfda56568cebedb6f
src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java
876e173ccbac04e4a06a245648c7c6af15eaaa92
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
c7de6e111300b009e1f9f430624a56100328184e
src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java
7eafe074b686d55ad96018006cf4acfa823513c3
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
ad9126c32893080e128d086ea3bfd7ad23d27b89
src/test/python/apache/aurora/client/cli/test_status.py
e531fa06e508d9792af51c62e67120c21baa7a81
src/test/python/apache/aurora/executor/test_thermos_executor.py
6c8ae1cf7fb578237708a954bd42162d66b39f4d
Diff: https://reviews.apache.org/r/28920/diff/
Testing
---
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69512 --- Ship it! Ship It! - Kevin Sweeney On Jan. 23, 2015, 4 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 23, 2015, 4 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh f692f025a7e5f2f0dddb7f6c81ea12fcb8272020 examples/vagrant/provision-dev-cluster.sh 40c165925c2110fb727c66ff5a34cf5ab8415343 examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 8428941c10641857a952f34df4e46a8fac5476a8 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/main/python/apache/aurora/config/schema/base.py d7897794c736778983d506c337a1392f3cc0cc20 src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py f29faf1d51baa4af66ad8c6579ffa354409e9536 src/main/python/apache/thermos/core/runner.py 41200bd5d74ee3239279567d79a5f48fb6af2e44 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 9fd188fb8f004d0a7664420bfda56568cebedb6f src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 7eafe074b686d55ad96018006cf4acfa823513c3 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java ad9126c32893080e128d086ea3bfd7ad23d27b89 src/test/python/apache/aurora/client/cli/test_status.py e531fa06e508d9792af51c62e67120c21baa7a81 src/test/python/apache/aurora/executor/test_thermos_executor.py 6c8ae1cf7fb578237708a954bd42162d66b39f4d Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 30010: [AURORA-184] Remove hardcoded 'host' and 'rack' limit constraints
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30010/ --- (Updated Jan. 23, 2015, 9:20 vorm.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-184 https://issues.apache.org/jira/browse/AURORA-184 Repository: aurora Description --- [AURORA-184] Remove hardcoded 'host' and 'rack' limit constraints This is the first step for AURORA-184, that removes the default hostrack limit constraints. The second step that's still missing would be to add s.th. like --default-constraints as start parameter to the scheduler. AURORA-174 could probably be closed with this?(since the rack limit constraint can be configured in the .aurora file) I can't really estimate the effect of my changes in StorageBackfillTestSchedulerThriftInterfaceTest, please have a closer look at the changes I did there. Since this is also my first code submit, comments about codestyleother bad habbits are very appreciated. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 Diff: https://reviews.apache.org/r/30010/diff/ Testing --- Added test for ConfigurationManager.hasName Added test testNoHostAndRackConstraintsAdded, that checks if the constraints are present Tested on vagrant devcluster to see if constraints are also gone in real life Thanks, Florian Pfeiffer
Re: Review Request 30010: [AURORA-184] Remove hardcoded 'host' and 'rack' limit constraints
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30010/#review69375 --- Ship it! Master (3fa004b) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 23, 2015, 9:20 a.m., Florian Pfeiffer wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30010/ --- (Updated Jan. 23, 2015, 9:20 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-184 https://issues.apache.org/jira/browse/AURORA-184 Repository: aurora Description --- [AURORA-184] Remove hardcoded 'host' and 'rack' limit constraints This is the first step for AURORA-184, that removes the default hostrack limit constraints. The second step that's still missing would be to add s.th. like --default-constraints as start parameter to the scheduler. AURORA-174 could probably be closed with this?(since the rack limit constraint can be configured in the .aurora file) I can't really estimate the effect of my changes in StorageBackfillTestSchedulerThriftInterfaceTest, please have a closer look at the changes I did there. Since this is also my first code submit, comments about codestyleother bad habbits are very appreciated. Diffs - src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 Diff: https://reviews.apache.org/r/30010/diff/ Testing --- Added test for ConfigurationManager.hasName Added test testNoHostAndRackConstraintsAdded, that checks if the constraints are present Tested on vagrant devcluster to see if constraints are also gone in real life Thanks, Florian Pfeiffer
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 22, 2015, 2:35 a.m., Kevin Sweeney wrote: docs/deploying-aurora-scheduler.md, line 163 https://reviews.apache.org/r/28920/diff/18/?file=823201#file823201line163 Philosophical question: if there's already a hard requirement that the container have Python 2.7 why not require that the executor be baked in as well? Maybe it's worth calling out as a TODO, but you don't have to answer it now. Steve Niemitz wrote: I think baking the executor into docker images is a recipe for disaster. Any time you upgraded aurora you'd need to then go update all containers with the new executor. Also, I don't like the idea of having to build specific aurora-isms into docker containers (I don't even really like requiring python, but that's unavoidable). Bill Farner wrote: I'm with Steve here. Forcing this seems overly restrictive. However, i would like to support the mode of operation you describe, Kevin. Kevin Sweeney wrote: Just seems weird to put partial requirements on the execution environment of the container we can run, which somewhat defeats the purpose of containers. Filed https://issues.apache.org/jira/browse/AURORA-1051 to explore removing this restriction. I'd love to see that as well. On Jan. 22, 2015, 2:35 a.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, lines 147-158 https://reviews.apache.org/r/28920/diff/18/?file=823211#file823211line147 I don't see a reason the executor can't do this itself, by reading the Container field in AssignedTask and environment variables. I'd prefer not to introduce a new channel to send configuration from the scheduler to the executor with this review. @wickman might be better able to answer whether this is feasible. Steve Niemitz wrote: Some of the things above can't be (easily) done here in the executor because it's already to late by the time code in the executor begins running. For example, logging is already initialized before any real app code runs. This has gone back and forth a few times now, and we landed on keeping anything docker specific out of the executor (which I like better anyways). There's been a good amount of conversation about this in reviews above. See below on this one. On Jan. 22, 2015, 2:35 a.m., Kevin Sweeney wrote: src/main/python/apache/thermos/core/runner.py, lines 625-632 https://reviews.apache.org/r/28920/diff/18/?file=823218#file823218line625 Naive question: since we have this block here can we drop the preamble from the scheduler? Steve Niemitz wrote: We need both, they work together to get the logs/sandbox into the right place, and allow the observer to pick it up. Also see below, this moved around a little bit so it's not in thermos anymore. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68845 --- On Jan. 16, 2015, 12:08 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 16, 2015, 12:08 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 23, 2015, 4:25 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Changes --- This includes a rebase from the latest master. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh f692f025a7e5f2f0dddb7f6c81ea12fcb8272020 examples/vagrant/provision-dev-cluster.sh 40c165925c2110fb727c66ff5a34cf5ab8415343 examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 8428941c10641857a952f34df4e46a8fac5476a8 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py f29faf1d51baa4af66ad8c6579ffa354409e9536 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 41200bd5d74ee3239279567d79a5f48fb6af2e44 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 9fd188fb8f004d0a7664420bfda56568cebedb6f src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 7eafe074b686d55ad96018006cf4acfa823513c3 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java ad9126c32893080e128d086ea3bfd7ad23d27b89 src/test/python/apache/aurora/client/cli/test_status.py e531fa06e508d9792af51c62e67120c21baa7a81 src/test/python/apache/aurora/executor/test_thermos_executor.py 6c8ae1cf7fb578237708a954bd42162d66b39f4d Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 22, 2015, 10:42 p.m., Brian Wickman wrote:
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java,
lines 153-158
https://reviews.apache.org/r/28920/diff/18/?file=823211#file823211line153
can't the stuff in DOCKER_COMMAND_PREFIX be accomplished with a
DockerSandboxProvider in the thermos executor?
similarly, DOCKER_COMMAND_SUFFIX should be unnecessary since this
information is available in the TaskConfig.
if both the above are true, then we don't need any of the changes to
CommandUtil.
Steve Niemitz wrote:
No unfortunatly, once the executor is started it's too late to change
some things, they need to be setup before the executor launches (see my
previous comment w/ Kevin). The suffix could be moved into the executor, but
again I'd like to keep docker specific logic out of there.
Brian Wickman wrote:
it's definitely not too late. and i disagree that we should keep docker
specific logic out of the executor. that's *exactly* where it should be.
afaict, the only thing affected is LogOptions.set_log_dir('.') in
thermos_executor_main. this can easily be changed to
LogOptions.set_log_dir(os.getenv('MESOS_DIRECTORY') or '.')
*ideally* in my opinion, the logical conclusion is creating a
thermos_docker_executor_main that wires things up in accordance to the docker
philosophy, then have a completely separate docker task factory within the
scheduler to construct these. but that might be a little more unsavory in
the short term than just having the thermos_executor that understands both
containerized and non-containerized modes. there is precedence for this
already (exactly why we have sandbox providers and task providers in the
first place -- to switch implementations entirely based upon incoming
TaskConfigs.)
Steve Niemitz wrote:
I can investiate moving it into the executor, but building an entirely
seperate executor for docker is more than I'm willing to take on at this
point.
Steve Niemitz wrote:
I need to think about this more, using your suggestion of
LogOptions.set_log_dir(os.getenv('MESOS_DIRECTORY') or '.') fails because the
directory doesn't yet exist (since the symlink hasn't yet been created). Its
possible we could defer setting up logging until the executor gets a task and
set everything up in aurora_executor, but I dont know what else that might
affect.
Ok I have something working here, MESOS_DIRECTORY into the logger doesn't work
since it doesn't exist yet (and I don't want the logger to create it), but
MESOS_SANDBOX does. That plus a little extra in the DefaultSandboxProvider and
removing references to . in thermos_executor_main seems to have done the
trick. Diff coming soon.
- Steve
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/#review69288
---
On Jan. 16, 2015, 12:08 a.m., Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/
---
(Updated Jan. 16, 2015, 12:08 a.m.)
Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner.
Bugs: AURORA-633
https://issues.apache.org/jira/browse/AURORA-633
Repository: aurora
Description
---
This change adds support for launching docker containers through aurora.
These changes are based off of the discussion in
https://issues.apache.org/jira/browse/AURORA-633
As of now, a special thermos_executor.sh script is needed to launch the
executor inside docker containers. A sample aurora file is in
examples/jobs/docker.
In addition, mesos-slave must be run with `--containerizers=docker,mesos`,
the example upstart config in examples/vagrant/upstart has been updated to
reflect this.
More information is in subsequent review request comments.
Diffs
-
Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc
api/src/main/thrift/org/apache/aurora/gen/api.thrift
08ba1cdf88b712de22c26c04443079282db59ef9
config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5
docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea
docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7
examples/jobs/docker/hello_docker.aurora PRE-CREATION
examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd
examples/vagrant/provision-dev-cluster.sh
7af4b52a6876268a97630279221bb98d9b04efad
examples/vagrant/upstart/aurora-scheduler.conf
788ec254270bca074dae91829c7f4fccdc8f8bb0
examples/vagrant/upstart/mesos-slave.conf
512ce7ecf34042ed68dda55efb2dd0415f8469db
