Review Request 32106: Changed the updater to not update an instance if only the job owner changes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32106/ --- Review request for Aurora and Bill Farner. Bugs: AURORA-1200 https://issues.apache.org/jira/browse/AURORA-1200 Repository: aurora Description --- Changed the updater to not update an instance if the job owner changes Diffs - src/main/java/org/apache/aurora/scheduler/updater/InstanceUpdater.java 14753cf5ef35d5133ca5029f3a465df884756070 src/test/java/org/apache/aurora/scheduler/updater/InstanceUpdaterTest.java 09c147e76d7c2c130a1fdd85459c45395fee7dde Diff: https://reviews.apache.org/r/32106/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 31338: Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/ --- (Updated March 3, 2015, 3:21 p.m.) Review request for Aurora, Jay Buffington and Bill Farner. Bugs: AURORA-1107 https://issues.apache.org/jira/browse/AURORA-1107 Repository: aurora Description --- Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run. This is the first portion of allowing per-job mounts, however, I wanted to get this out first since more people want it. I'll implement per-job mounts in a future review. Diffs (updated) - api/src/main/thrift/org/apache/aurora/gen/api.thrift a5198dfe1c56e05d50e32dd14222a984515c0d07 config/checkstyle/checkstyle.xml 580d9d3737c0f52da2d33f5b99ef9406b55c5807 docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 3239eaa139e35e8c3acdacf6375f492de2b5bfee src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 24b61c1e4f615295acf28d904588e1512972d3f4 src/main/java/org/apache/aurora/scheduler/app/VolumeParser.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java baacb71403d55c5b90fc11cb2a23f552a32e8ba5 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5340d651b298ec8aa079e73d6d2f652fdf876293 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 6575b7d420f17ec68d6e2a83e7b380f684577d4f src/test/java/org/apache/aurora/scheduler/app/VolumeParserTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 444d6d3fdaf86eb84612f846eaa326eb75c49898 src/test/java/org/apache/aurora/scheduler/mesos/TaskExecutors.java efe62ceb502ead88a2f0cd6d09a76664e465d9bc Diff: https://reviews.apache.org/r/31338/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 31423: Stop the announcer and status checkers before starting to kill the runners
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31423/#review74958 --- Is there anything else here? Is this ready to be merged? - Steve Niemitz On Feb. 26, 2015, 3:20 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31423/ --- (Updated Feb. 26, 2015, 3:20 p.m.) Review request for Aurora, Brian Wickman and Zameer Manji. Repository: aurora Description --- Stop the announcer and status checkers before starting to kill the runners. This allows the task to be removed from the ZK ensemble before it begins getting killed. The delay can be significant if the task takes some time to shutdown, and during the time it stops responding to requests. Diffs - src/main/python/apache/aurora/executor/aurora_executor.py 9c0282392dbb9cca308baf47adc1750c1f5cacc6 src/test/python/apache/aurora/executor/BUILD 2ee9b1233e9db47455ddffbc48691d379222 src/test/python/apache/aurora/executor/test_thermos_executor.py 8dbfb1db5eb7a6548820ff7cf82a9c7092f61d28 Diff: https://reviews.apache.org/r/31423/diff/ Testing --- We're now running this in our production environments. Watching ZK, I can confirm that the nodes are removed before process shutdown begins. Watching the executor log also confirms this. I couldn't observe any other side effects either. Thanks, Steve Niemitz
Re: Review Request 31338: Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/ --- (Updated March 2, 2015, 9:03 p.m.) Review request for Aurora, Jay Buffington and Bill Farner. Bugs: AURORA-1107 https://issues.apache.org/jira/browse/AURORA-1107 Repository: aurora Description --- Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run. This is the first portion of allowing per-job mounts, however, I wanted to get this out first since more people want it. I'll implement per-job mounts in a future review. Diffs (updated) - api/src/main/thrift/org/apache/aurora/gen/api.thrift 6f6124a4c844a1abcf07401d80c3e50eb8b4 config/checkstyle/checkstyle.xml 580d9d3737c0f52da2d33f5b99ef9406b55c5807 docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java bacfbfeb237ecddf82f58679e05be012c5214e61 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 24b61c1e4f615295acf28d904588e1512972d3f4 src/main/java/org/apache/aurora/scheduler/app/VolumeParser.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java baacb71403d55c5b90fc11cb2a23f552a32e8ba5 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5340d651b298ec8aa079e73d6d2f652fdf876293 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 6575b7d420f17ec68d6e2a83e7b380f684577d4f src/test/java/org/apache/aurora/scheduler/app/VolumeParserTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 444d6d3fdaf86eb84612f846eaa326eb75c49898 src/test/java/org/apache/aurora/scheduler/mesos/TaskExecutors.java efe62ceb502ead88a2f0cd6d09a76664e465d9bc Diff: https://reviews.apache.org/r/31338/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 31338: Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run.
On March 2, 2015, 6:56 p.m., Bill Farner wrote: src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java, line 133 https://reviews.apache.org/r/31338/diff/8/?file=882131#file882131line133 I think you can drop this since it's == `Resources.NONE` (default). Done On March 2, 2015, 6:56 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java, line 137 https://reviews.apache.org/r/31338/diff/8/?file=882134#file882134line137 remove empty line Done On March 2, 2015, 6:56 p.m., Bill Farner wrote: src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java, lines 216-222 https://reviews.apache.org/r/31338/diff/8/?file=882138#file882138line216 This can collapse to: ``` assertTrue(taskInfo.getExecutor().getContainer().getVolumesList().contains(expected)); ``` Done - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/#review74782 --- On March 2, 2015, 4:59 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/ --- (Updated March 2, 2015, 4:59 p.m.) Review request for Aurora, Jay Buffington and Bill Farner. Bugs: AURORA-1107 https://issues.apache.org/jira/browse/AURORA-1107 Repository: aurora Description --- Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run. This is the first portion of allowing per-job mounts, however, I wanted to get this out first since more people want it. I'll implement per-job mounts in a future review. Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift 6f6124a4c844a1abcf07401d80c3e50eb8b4 config/checkstyle/checkstyle.xml 580d9d3737c0f52da2d33f5b99ef9406b55c5807 docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java bacfbfeb237ecddf82f58679e05be012c5214e61 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 24b61c1e4f615295acf28d904588e1512972d3f4 src/main/java/org/apache/aurora/scheduler/app/VolumeParser.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java baacb71403d55c5b90fc11cb2a23f552a32e8ba5 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5340d651b298ec8aa079e73d6d2f652fdf876293 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 6575b7d420f17ec68d6e2a83e7b380f684577d4f src/test/java/org/apache/aurora/scheduler/app/VolumeParserTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 444d6d3fdaf86eb84612f846eaa326eb75c49898 src/test/java/org/apache/aurora/scheduler/mesos/TaskExecutors.java efe62ceb502ead88a2f0cd6d09a76664e465d9bc Diff: https://reviews.apache.org/r/31338/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 31338: Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/#review74752 --- @ReviewBot retry - Steve Niemitz On March 2, 2015, 4:59 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/ --- (Updated March 2, 2015, 4:59 p.m.) Review request for Aurora, Jay Buffington and Bill Farner. Bugs: AURORA-1107 https://issues.apache.org/jira/browse/AURORA-1107 Repository: aurora Description --- Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run. This is the first portion of allowing per-job mounts, however, I wanted to get this out first since more people want it. I'll implement per-job mounts in a future review. Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift 6f6124a4c844a1abcf07401d80c3e50eb8b4 config/checkstyle/checkstyle.xml 580d9d3737c0f52da2d33f5b99ef9406b55c5807 docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java bacfbfeb237ecddf82f58679e05be012c5214e61 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 24b61c1e4f615295acf28d904588e1512972d3f4 src/main/java/org/apache/aurora/scheduler/app/VolumeParser.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java baacb71403d55c5b90fc11cb2a23f552a32e8ba5 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5340d651b298ec8aa079e73d6d2f652fdf876293 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 6575b7d420f17ec68d6e2a83e7b380f684577d4f src/test/java/org/apache/aurora/scheduler/app/VolumeParserTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 444d6d3fdaf86eb84612f846eaa326eb75c49898 src/test/java/org/apache/aurora/scheduler/mesos/TaskExecutors.java efe62ceb502ead88a2f0cd6d09a76664e465d9bc Diff: https://reviews.apache.org/r/31338/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 31338: Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/ --- (Updated Feb. 27, 2015, 8:19 p.m.) Review request for Aurora, Jay Buffington and Bill Farner. Bugs: AURORA-1107 https://issues.apache.org/jira/browse/AURORA-1107 Repository: aurora Description --- Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run. This is the first portion of allowing per-job mounts, however, I wanted to get this out first since more people want it. I'll implement per-job mounts in a future review. Diffs (updated) - api/src/main/thrift/org/apache/aurora/gen/api.thrift 6f6124a4c844a1abcf07401d80c3e50eb8b4 config/checkstyle/checkstyle.xml 580d9d3737c0f52da2d33f5b99ef9406b55c5807 docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java bacfbfeb237ecddf82f58679e05be012c5214e61 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 24b61c1e4f615295acf28d904588e1512972d3f4 src/main/java/org/apache/aurora/scheduler/app/VolumeParser.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java baacb71403d55c5b90fc11cb2a23f552a32e8ba5 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5340d651b298ec8aa079e73d6d2f652fdf876293 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 6575b7d420f17ec68d6e2a83e7b380f684577d4f src/test/java/org/apache/aurora/scheduler/app/VolumeParserTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 444d6d3fdaf86eb84612f846eaa326eb75c49898 src/test/java/org/apache/aurora/scheduler/mesos/TaskExecutors.java efe62ceb502ead88a2f0cd6d09a76664e465d9bc Diff: https://reviews.apache.org/r/31338/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 31338: Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run.
On Feb. 27, 2015, 5:01 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java, line 36 https://reviews.apache.org/r/31338/diff/4/?file=876495#file876495line36 should this be private? I wanted it to be, but got style checker violations when making it private. Same reason the Builder ctor is public as well. On Feb. 27, 2015, 5:01 p.m., Bill Farner wrote: src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java, lines 215-227 https://reviews.apache.org/r/31338/diff/4/?file=876499#file876499line215 Can this instead be an exact comparison of ListVolume? Presumably we should fail if other volumes show up. ``` assertEquals( ImmutableList.of(expected), taskInfo.getExecutor().getContainer().getVolumesList()); ``` I think checking only this specifically is a more robust test. There are other things we may want to mount in by default in the future (for example, we already mount the thermos run directory), and testing for all of them here would make the test to broad. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/#review74507 --- On Feb. 25, 2015, 10:35 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/ --- (Updated Feb. 25, 2015, 10:35 p.m.) Review request for Aurora, Jay Buffington and Bill Farner. Bugs: AURORA-1107 https://issues.apache.org/jira/browse/AURORA-1107 Repository: aurora Description --- Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run. This is the first portion of allowing per-job mounts, however, I wanted to get this out first since more people want it. I'll implement per-job mounts in a future review. Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift 6f6124a4c844a1abcf07401d80c3e50eb8b4 config/checkstyle/checkstyle.xml 580d9d3737c0f52da2d33f5b99ef9406b55c5807 docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java bacfbfeb237ecddf82f58679e05be012c5214e61 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 24b61c1e4f615295acf28d904588e1512972d3f4 src/main/java/org/apache/aurora/scheduler/app/VolumeParser.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java baacb71403d55c5b90fc11cb2a23f552a32e8ba5 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5340d651b298ec8aa079e73d6d2f652fdf876293 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 6575b7d420f17ec68d6e2a83e7b380f684577d4f src/test/java/org/apache/aurora/scheduler/app/VolumeParserTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 444d6d3fdaf86eb84612f846eaa326eb75c49898 src/test/java/org/apache/aurora/scheduler/mesos/TaskExecutors.java efe62ceb502ead88a2f0cd6d09a76664e465d9bc Diff: https://reviews.apache.org/r/31338/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 31338: Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/ --- (Updated Feb. 27, 2015, 11:16 p.m.) Review request for Aurora, Jay Buffington and Bill Farner. Bugs: AURORA-1107 https://issues.apache.org/jira/browse/AURORA-1107 Repository: aurora Description --- Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run. This is the first portion of allowing per-job mounts, however, I wanted to get this out first since more people want it. I'll implement per-job mounts in a future review. Diffs (updated) - api/src/main/thrift/org/apache/aurora/gen/api.thrift 6f6124a4c844a1abcf07401d80c3e50eb8b4 config/checkstyle/checkstyle.xml 580d9d3737c0f52da2d33f5b99ef9406b55c5807 docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java bacfbfeb237ecddf82f58679e05be012c5214e61 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 24b61c1e4f615295acf28d904588e1512972d3f4 src/main/java/org/apache/aurora/scheduler/app/VolumeParser.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java baacb71403d55c5b90fc11cb2a23f552a32e8ba5 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5340d651b298ec8aa079e73d6d2f652fdf876293 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 6575b7d420f17ec68d6e2a83e7b380f684577d4f src/test/java/org/apache/aurora/scheduler/app/VolumeParserTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 444d6d3fdaf86eb84612f846eaa326eb75c49898 src/test/java/org/apache/aurora/scheduler/mesos/TaskExecutors.java efe62ceb502ead88a2f0cd6d09a76664e465d9bc Diff: https://reviews.apache.org/r/31338/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 31338: Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run.
On Feb. 25, 2015, 10:06 p.m., Joshua Cohen wrote: src/test/java/org/apache/aurora/scheduler/app/VolumeParserTest.java, lines 44-45 https://reviews.apache.org/r/31338/diff/2/?file=876074#file876074line44 this should be unnecessary, we will have thrown on the previous line? Also can you add a test for too many parts as well (e.g. foo:bar:baz:quux)? I added these in to stop some complaints about unused variables, but in retrospect I could have just called vp.doParse(...) and not assigned it. I'll change these. On Feb. 25, 2015, 10:06 p.m., Joshua Cohen wrote: src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java, line 214 https://reviews.apache.org/r/31338/diff/2/?file=876075#file876075line214 move to next line. Also is this potentially fragile to assume the global mount will be the second mount in the list? It'd be slightly more robust to iterate the mounts and ensure the mount we expect is *somewhere* in the list? Good point on the ordering, it shouldn't matter. I'll change it. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/#review74103 --- On Feb. 25, 2015, 7:15 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/ --- (Updated Feb. 25, 2015, 7:15 p.m.) Review request for Aurora, Jay Buffington and Bill Farner. Bugs: AURORA-1107 https://issues.apache.org/jira/browse/AURORA-1107 Repository: aurora Description --- Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run. This is the first portion of allowing per-job mounts, however, I wanted to get this out first since more people want it. I'll implement per-job mounts in a future review. Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift 6f6124a4c844a1abcf07401d80c3e50eb8b4 config/checkstyle/checkstyle.xml 580d9d3737c0f52da2d33f5b99ef9406b55c5807 docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java bacfbfeb237ecddf82f58679e05be012c5214e61 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 24b61c1e4f615295acf28d904588e1512972d3f4 src/main/java/org/apache/aurora/scheduler/app/VolumeParser.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java baacb71403d55c5b90fc11cb2a23f552a32e8ba5 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5340d651b298ec8aa079e73d6d2f652fdf876293 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 6575b7d420f17ec68d6e2a83e7b380f684577d4f src/test/java/org/apache/aurora/scheduler/app/VolumeParserTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 444d6d3fdaf86eb84612f846eaa326eb75c49898 src/test/java/org/apache/aurora/scheduler/mesos/TaskExecutors.java efe62ceb502ead88a2f0cd6d09a76664e465d9bc Diff: https://reviews.apache.org/r/31338/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 31423: Stop the announcer and status checkers before starting to kill the runners
On Feb. 26, 2015, 3:38 a.m., Joshua Cohen wrote: src/test/python/apache/aurora/executor/test_thermos_executor.py, lines 384-386 https://reviews.apache.org/r/31423/diff/2/?file=877156#file877156line384 You should be able to use contextlib.nested for this: with contextlib.nested( temporary_dir(), mock.patch.object(...), mock.patch.object(...)) as ( checkpoint_root, status_check_stop, runner_stop): # use all the things Alternately, I think you can just wrap the with in parens as well, since we're pinned to python2.7 now (though a quick scan of the code doesn't reveal any place that we do that, but does show lots of places where we use contextlib.nested). I tried parens and they don't work with with for whatever reason (syntax error). I tried to avoid contextlib.nested because it's deprecated as of 2.7, but I can use that if you guys think its ok. On Feb. 26, 2015, 3:38 a.m., Joshua Cohen wrote: src/test/python/apache/aurora/executor/test_thermos_executor.py, line 398 https://reviews.apache.org/r/31423/diff/2/?file=877156#file877156line398 Is it worth being explicit and passing any_order=False (even though it's the default), just to make it clear that the order is important? I had to look it up to see what the behavior was, maybe for people more familiar with python/mock would know right away, but it's not abundantly clear. :thumbsup: - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31423/#review74202 --- On Feb. 26, 2015, 2:14 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31423/ --- (Updated Feb. 26, 2015, 2:14 a.m.) Review request for Aurora, Brian Wickman and Zameer Manji. Repository: aurora Description --- Stop the announcer and status checkers before starting to kill the runners. This allows the task to be removed from the ZK ensemble before it begins getting killed. The delay can be significant if the task takes some time to shutdown, and during the time it stops responding to requests. Diffs - src/main/python/apache/aurora/executor/aurora_executor.py 9c0282392dbb9cca308baf47adc1750c1f5cacc6 src/test/python/apache/aurora/executor/BUILD 2ee9b1233e9db47455ddffbc48691d379222 src/test/python/apache/aurora/executor/test_thermos_executor.py 8dbfb1db5eb7a6548820ff7cf82a9c7092f61d28 Diff: https://reviews.apache.org/r/31423/diff/ Testing --- We're now running this in our production environments. Watching ZK, I can confirm that the nodes are removed before process shutdown begins. Watching the executor log also confirms this. I couldn't observe any other side effects either. Thanks, Steve Niemitz
Re: Review Request 31338: Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/ --- (Updated Feb. 25, 2015, 10:35 p.m.) Review request for Aurora, Jay Buffington and Bill Farner. Changes --- That's what I get for not running tests on trivial changes... Bugs: AURORA-1107 https://issues.apache.org/jira/browse/AURORA-1107 Repository: aurora Description --- Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run. This is the first portion of allowing per-job mounts, however, I wanted to get this out first since more people want it. I'll implement per-job mounts in a future review. Diffs (updated) - api/src/main/thrift/org/apache/aurora/gen/api.thrift 6f6124a4c844a1abcf07401d80c3e50eb8b4 config/checkstyle/checkstyle.xml 580d9d3737c0f52da2d33f5b99ef9406b55c5807 docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java bacfbfeb237ecddf82f58679e05be012c5214e61 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 24b61c1e4f615295acf28d904588e1512972d3f4 src/main/java/org/apache/aurora/scheduler/app/VolumeParser.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java baacb71403d55c5b90fc11cb2a23f552a32e8ba5 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5340d651b298ec8aa079e73d6d2f652fdf876293 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 6575b7d420f17ec68d6e2a83e7b380f684577d4f src/test/java/org/apache/aurora/scheduler/app/VolumeParserTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 444d6d3fdaf86eb84612f846eaa326eb75c49898 src/test/java/org/apache/aurora/scheduler/mesos/TaskExecutors.java efe62ceb502ead88a2f0cd6d09a76664e465d9bc Diff: https://reviews.apache.org/r/31338/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 31338: Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run.
On Feb. 24, 2015, 6:01 a.m., Joshua Cohen wrote:
Steve Niemitz wrote:
I'm not a big fan of how the parsing works here either. I was thinking
about this last night, I think I have a better plan here. Lemme know what
you think.
I already want to add volume support per-job at some point, so I propose
adding the needed thrift objects to api.thrift (Volume, Mode enum), and
converting the command line into those objects in ExecutorSettings. That
would then let me reuse the volume-adding code in MesosTaskFactory for normal
volumes in the future.
Re: command line, I chose the format there because it's the same as the
docker command line. What about making a new parser to parse that into the
above mentioned thrift objects?
Stephan Erb wrote:
How about using the same format and argument name as Mesos'
`--default_container_info`:
{
type: MESOS,
volumes: [
{
host_path: ./.private/tmp,
container_path: /tmp,
mode: RW
}
]
}
See https://mesos.apache.org/documentation/latest/configuration/
JSON on the command line is an escaping nightmare IMO, and very verbose to
specify 2 (or 3) strings for this use case.
- Steve
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31338/#review73755
---
On Feb. 24, 2015, 2:07 a.m., Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31338/
---
(Updated Feb. 24, 2015, 2:07 a.m.)
Review request for Aurora, Jay Buffington and Bill Farner.
Bugs: AURORA-1107
https://issues.apache.org/jira/browse/AURORA-1107
Repository: aurora
Description
---
Added a command line flag, -global_container_mounts, to allow mounting paths
from the slaves into the (docker) containers they run.
This is the first portion of allowing per-job mounts, however, I wanted to
get this out first since more people want it. I'll implement per-job mounts
in a future review.
Diffs
-
docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f
src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java
bacfbfeb237ecddf82f58679e05be012c5214e61
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java
24b61c1e4f615295acf28d904588e1512972d3f4
src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java
baacb71403d55c5b90fc11cb2a23f552a32e8ba5
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
5340d651b298ec8aa079e73d6d2f652fdf876293
src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java
6575b7d420f17ec68d6e2a83e7b380f684577d4f
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
444d6d3fdaf86eb84612f846eaa326eb75c49898
src/test/java/org/apache/aurora/scheduler/mesos/TaskExecutors.java
efe62ceb502ead88a2f0cd6d09a76664e465d9bc
Diff: https://reviews.apache.org/r/31338/diff/
Testing
---
Thanks,
Steve Niemitz
Re: Review Request 31338: Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run.
On Feb. 24, 2015, 6:01 a.m., Joshua Cohen wrote:
Steve Niemitz wrote:
I'm not a big fan of how the parsing works here either. I was thinking
about this last night, I think I have a better plan here. Lemme know what
you think.
I already want to add volume support per-job at some point, so I propose
adding the needed thrift objects to api.thrift (Volume, Mode enum), and
converting the command line into those objects in ExecutorSettings. That
would then let me reuse the volume-adding code in MesosTaskFactory for normal
volumes in the future.
Re: command line, I chose the format there because it's the same as the
docker command line. What about making a new parser to parse that into the
above mentioned thrift objects?
Stephan Erb wrote:
How about using the same format and argument name as Mesos'
`--default_container_info`:
{
type: MESOS,
volumes: [
{
host_path: ./.private/tmp,
container_path: /tmp,
mode: RW
}
]
}
See https://mesos.apache.org/documentation/latest/configuration/
Steve Niemitz wrote:
JSON on the command line is an escaping nightmare IMO, and very verbose
to specify 2 (or 3) strings for this use case.
Bill Farner wrote:
JSON on the command line also makes me uneasy, but i don't think we have
precedent for command line entities with this many knobs (3 are obvious so
far).
How about 2 approaches as straw men:
`-volume_mounts_read_only` and `-volume_mounts_read_write`, both
key-value mappings
`-volume_mounts_config`, path to a JSON file with entities similar to
what Stephan sketched above
The first approach has the upside that it fits with the status quo, but
it cannot be easily extended if we ever want more attributes. The second
would be the first config file we use, but is more future-proof.
Joshua Cohen wrote:
I think a single arg is easiest to reason about. I think I'm ok with
Steve's follow up suggestion to create the thrift types that will be needed
for per-job mounts now and use them to aid in arg parsing? That said, I'm not
sure we should enforce the docker mount format since in theory these mounts
can/will apply to other containers where they may not make as much sense.
Steve Niemitz wrote:
I think it's simple enough it'd make sense for any container system.
host:container:mode is pretty generic. I'm also not as big of a fan about
multiple args, and a default of RO if not specified seems reasonable to me.
Bill Farner wrote:
Works for me, but -1 to defaulting. I much prefer explicit over
implicit, and this way it's easy to see how to turn a read-only mount into
read-write.
Cool with me. Ok I think we're all on the same page here. I'll submit an
updated patch in a little bit.
- Steve
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31338/#review73755
---
On Feb. 24, 2015, 2:07 a.m., Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31338/
---
(Updated Feb. 24, 2015, 2:07 a.m.)
Review request for Aurora, Jay Buffington and Bill Farner.
Bugs: AURORA-1107
https://issues.apache.org/jira/browse/AURORA-1107
Repository: aurora
Description
---
Added a command line flag, -global_container_mounts, to allow mounting paths
from the slaves into the (docker) containers they run.
This is the first portion of allowing per-job mounts, however, I wanted to
get this out first since more people want it. I'll implement per-job mounts
in a future review.
Diffs
-
docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f
src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java
bacfbfeb237ecddf82f58679e05be012c5214e61
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java
24b61c1e4f615295acf28d904588e1512972d3f4
src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java
baacb71403d55c5b90fc11cb2a23f552a32e8ba5
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
5340d651b298ec8aa079e73d6d2f652fdf876293
src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java
6575b7d420f17ec68d6e2a83e7b380f684577d4f
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
444d6d3fdaf86eb84612f846eaa326eb75c49898
src/test/java/org/apache/aurora/scheduler/mesos/TaskExecutors.java
efe62ceb502ead88a2f0cd6d09a76664e465d9bc
Diff: https://reviews.apache.org/r/31338/diff
Review Request 31338: Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31338/ --- Review request for Aurora, Jay Buffington and Bill Farner. Bugs: AURORA-1107 https://issues.apache.org/jira/browse/AURORA-1107 Repository: aurora Description --- Added a command line flag, -global_container_mounts, to allow mounting paths from the slaves into the (docker) containers they run. This is the first portion of allowing per-job mounts, however, I wanted to get this out first since more people want it. I'll implement per-job mounts in a future review. Diffs - docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java bacfbfeb237ecddf82f58679e05be012c5214e61 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 24b61c1e4f615295acf28d904588e1512972d3f4 src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java baacb71403d55c5b90fc11cb2a23f552a32e8ba5 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5340d651b298ec8aa079e73d6d2f652fdf876293 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 6575b7d420f17ec68d6e2a83e7b380f684577d4f src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 444d6d3fdaf86eb84612f846eaa326eb75c49898 src/test/java/org/apache/aurora/scheduler/mesos/TaskExecutors.java efe62ceb502ead88a2f0cd6d09a76664e465d9bc Diff: https://reviews.apache.org/r/31338/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 30471: Revert backwards incomaptible chnages to GC ExecutorInfo.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30471/#review70447 --- Ship it! This looks good to me. As long as the default value is true there's no changes to the functionality my Docker implementation needs. I'm still curious why I'm not running into this issues though. - Steve Niemitz On Jan. 30, 2015, 11:24 p.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30471/ --- (Updated Jan. 30, 2015, 11:24 p.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-1076 https://issues.apache.org/jira/browse/AURORA-1076 Repository: aurora Description --- The Docker patch (7ba6226) started to set the shell attribute of of ExecutorInfo. This breaks backwards compatability for the GC executor as it will continue to have the same id but different info now. In addition it is not necessary to set this value because it defaults to true. Diffs - src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java PRE-CREATION Diff: https://reviews.apache.org/r/30471/diff/ Testing --- ./gradlew build -Pq Thanks, Zameer Manji
Re: Review Request 30384: Added TellApart to list of aurora users.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30384/ --- (Updated Jan. 28, 2015, 11:38 p.m.) Review request for Aurora and Bill Farner. Changes --- So many more Ss. Repository: aurora Description --- Added TellApart to list of aurora users. Diffs (updated) - README.md 61e0253b1f7a3958c1e8444a071d8364b4164fab Diff: https://reviews.apache.org/r/30384/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 30384: Added TellApart to list of aurora users.
On Jan. 28, 2015, 11:31 p.m., Bill Farner wrote: README.md, line 65 https://reviews.apache.org/r/30384/diff/1/?file=839312#file839312line65 http One of these things is not like the others :-) Our https site is just a 301 to http. I can keep it https if you want though. :) - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30384/#review70107 --- On Jan. 28, 2015, 11:03 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30384/ --- (Updated Jan. 28, 2015, 11:03 p.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- Added TellApart to list of aurora users. Diffs - README.md 61e0253b1f7a3958c1e8444a071d8364b4164fab Diff: https://reviews.apache.org/r/30384/diff/ Testing --- Thanks, Steve Niemitz
Review Request 30384: Added TellApart to list of aurora users.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30384/ --- Review request for Aurora and Bill Farner. Repository: aurora Description --- Added TellApart to list of aurora users. Diffs - README.md 61e0253b1f7a3958c1e8444a071d8364b4164fab Diff: https://reviews.apache.org/r/30384/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29827: End to end tests for docker in aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29827/ --- (Updated Jan. 28, 2015, 6:56 p.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- This adds an end to end test for docker. Diffs (updated) - src/test/sh/org/apache/aurora/e2e/Dockerfile PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_docker.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_docker_updated.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 45da754341de52759d05a8960a9a978111f1e415 Diff: https://reviews.apache.org/r/29827/diff/ Testing --- Thanks, Steve Niemitz
Review Request 30331: Fixed python tests hanging
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30331/ --- Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- Fixed python tests hanging Diffs - src/test/python/apache/aurora/executor/test_thermos_executor.py e6191f3e7775663ad4a75546b5e86289bb73e989 Diff: https://reviews.apache.org/r/30331/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 30331: Fixed python tests hanging
On Jan. 27, 2015, 10:32 p.m., Kevin Sweeney wrote: Any idea which change introduced this breakage? Can you elaborate in the description? Sorry yeah, I was writing up an email to the dev list. I'll just post here instead. I actually was the one who broke this with a tiny change in [28920](https://reviews.apache.org/r/28920/#review69480). The unit test didn't have the job set on the AssignedTask object, so it failed trying to read it when creating the sandbox. The actual error the test was failing with was: ``` FATAL] Task killed during initialization. Traceback (most recent call last): File /tmp/tmpOboqH_/.deps/twitter.common.exceptions-0.3.2-py2-none-any.whl/twitter/common/exceptions/__init__.py, line 126, in _excepting_run self.__real_run(*args, **kw) File /tmp/tmpOboqH_/.deps/twitter.common.concurrent-0.3.2-py2-none-any.whl/twitter/common/concurrent/deferred.py, line 43, in run self._closure() File /tmp/tmpOboqH_/apache/aurora/executor/aurora_executor.py, line 252, in lambda defer(lambda: self._run(driver, assigned_task)) File /tmp/tmpOboqH_/apache/aurora/executor/aurora_executor.py, line 96, in _run self._runner = self._runner_provider.from_assigned_task(assigned_task, self._sandbox) File /tmp/tmpOboqH_/apache/aurora/executor/thermos_task_runner.py, line 382, in from_assigned_task role = self._get_role(assigned_task) File /tmp/tmpOboqH_/apache/aurora/executor/thermos_task_runner.py, line 378, in _get_role return None if assigned_task.task.container.docker else assigned_task.task.job.role AttributeError: 'NoneType' object has no attribute 'role' ``` however, it never got bubbled back to the unit test, and the test instead hung at `proxy_driver.wait_stopped()` - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30331/#review69902 --- On Jan. 27, 2015, 10:28 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30331/ --- (Updated Jan. 27, 2015, 10:28 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- Fixed python tests hanging Diffs - src/test/python/apache/aurora/executor/test_thermos_executor.py e6191f3e7775663ad4a75546b5e86289bb73e989 Diff: https://reviews.apache.org/r/30331/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 30331: Fixed python tests hanging
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30331/ --- (Updated Jan. 28, 2015, 12:13 a.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Changes --- +ticket Bugs: AURORA-1067 https://issues.apache.org/jira/browse/AURORA-1067 Repository: aurora Description --- Fixed python tests hanging Diffs - src/test/python/apache/aurora/executor/test_thermos_executor.py e6191f3e7775663ad4a75546b5e86289bb73e989 Diff: https://reviews.apache.org/r/30331/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 30331: Fixed python tests hanging
On Jan. 27, 2015, 10:32 p.m., Kevin Sweeney wrote: Any idea which change introduced this breakage? Can you elaborate in the description? Steve Niemitz wrote: Sorry yeah, I was writing up an email to the dev list. I'll just post here instead. I actually was the one who broke this with a tiny change in [28920](https://reviews.apache.org/r/28920/#review69480). The unit test didn't have the job set on the AssignedTask object, so it failed trying to read it when creating the sandbox. The actual error the test was failing with was: ``` FATAL] Task killed during initialization. Traceback (most recent call last): File /tmp/tmpOboqH_/.deps/twitter.common.exceptions-0.3.2-py2-none-any.whl/twitter/common/exceptions/__init__.py, line 126, in _excepting_run self.__real_run(*args, **kw) File /tmp/tmpOboqH_/.deps/twitter.common.concurrent-0.3.2-py2-none-any.whl/twitter/common/concurrent/deferred.py, line 43, in run self._closure() File /tmp/tmpOboqH_/apache/aurora/executor/aurora_executor.py, line 252, in lambda defer(lambda: self._run(driver, assigned_task)) File /tmp/tmpOboqH_/apache/aurora/executor/aurora_executor.py, line 96, in _run self._runner = self._runner_provider.from_assigned_task(assigned_task, self._sandbox) File /tmp/tmpOboqH_/apache/aurora/executor/thermos_task_runner.py, line 382, in from_assigned_task role = self._get_role(assigned_task) File /tmp/tmpOboqH_/apache/aurora/executor/thermos_task_runner.py, line 378, in _get_role return None if assigned_task.task.container.docker else assigned_task.task.job.role AttributeError: 'NoneType' object has no attribute 'role' ``` however, it never got bubbled back to the unit test, and the test instead hung at `proxy_driver.wait_stopped()` Also I think as a TOOD we should implement something similar to https://reviews.apache.org/r/28486 when creating and initalizing the task runner provider. I'll add a TODO and create a jira ticket for that. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30331/#review69902 --- On Jan. 27, 2015, 10:28 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30331/ --- (Updated Jan. 27, 2015, 10:28 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- Fixed python tests hanging Diffs - src/test/python/apache/aurora/executor/test_thermos_executor.py e6191f3e7775663ad4a75546b5e86289bb73e989 Diff: https://reviews.apache.org/r/30331/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29827: End to end tests for docker in aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29827/ --- (Updated Jan. 27, 2015, 4:58 p.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- This adds an end to end test for docker. Diffs (updated) - src/test/sh/org/apache/aurora/e2e/Dockerfile PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_docker.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_docker_updated.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 45da754341de52759d05a8960a9a978111f1e415 Diff: https://reviews.apache.org/r/29827/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69439 --- @ReviewBot retry - Steve Niemitz On Jan. 23, 2015, 5:23 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 23, 2015, 5:23 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh f692f025a7e5f2f0dddb7f6c81ea12fcb8272020 examples/vagrant/provision-dev-cluster.sh 40c165925c2110fb727c66ff5a34cf5ab8415343 examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 8428941c10641857a952f34df4e46a8fac5476a8 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py f29faf1d51baa4af66ad8c6579ffa354409e9536 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 41200bd5d74ee3239279567d79a5f48fb6af2e44 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 9fd188fb8f004d0a7664420bfda56568cebedb6f src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 7eafe074b686d55ad96018006cf4acfa823513c3 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java ad9126c32893080e128d086ea3bfd7ad23d27b89 src/test/python/apache/aurora/client/cli/test_status.py e531fa06e508d9792af51c62e67120c21baa7a81 src/test/python/apache/aurora/executor/test_thermos_executor.py 6c8ae1cf7fb578237708a954bd42162d66b39f4d Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 22, 2015, 2:35 a.m., Kevin Sweeney wrote: docs/deploying-aurora-scheduler.md, line 163 https://reviews.apache.org/r/28920/diff/18/?file=823201#file823201line163 Philosophical question: if there's already a hard requirement that the container have Python 2.7 why not require that the executor be baked in as well? Maybe it's worth calling out as a TODO, but you don't have to answer it now. Steve Niemitz wrote: I think baking the executor into docker images is a recipe for disaster. Any time you upgraded aurora you'd need to then go update all containers with the new executor. Also, I don't like the idea of having to build specific aurora-isms into docker containers (I don't even really like requiring python, but that's unavoidable). Bill Farner wrote: I'm with Steve here. Forcing this seems overly restrictive. However, i would like to support the mode of operation you describe, Kevin. Kevin Sweeney wrote: Just seems weird to put partial requirements on the execution environment of the container we can run, which somewhat defeats the purpose of containers. Filed https://issues.apache.org/jira/browse/AURORA-1051 to explore removing this restriction. I'd love to see that as well. On Jan. 22, 2015, 2:35 a.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, lines 147-158 https://reviews.apache.org/r/28920/diff/18/?file=823211#file823211line147 I don't see a reason the executor can't do this itself, by reading the Container field in AssignedTask and environment variables. I'd prefer not to introduce a new channel to send configuration from the scheduler to the executor with this review. @wickman might be better able to answer whether this is feasible. Steve Niemitz wrote: Some of the things above can't be (easily) done here in the executor because it's already to late by the time code in the executor begins running. For example, logging is already initialized before any real app code runs. This has gone back and forth a few times now, and we landed on keeping anything docker specific out of the executor (which I like better anyways). There's been a good amount of conversation about this in reviews above. See below on this one. On Jan. 22, 2015, 2:35 a.m., Kevin Sweeney wrote: src/main/python/apache/thermos/core/runner.py, lines 625-632 https://reviews.apache.org/r/28920/diff/18/?file=823218#file823218line625 Naive question: since we have this block here can we drop the preamble from the scheduler? Steve Niemitz wrote: We need both, they work together to get the logs/sandbox into the right place, and allow the observer to pick it up. Also see below, this moved around a little bit so it's not in thermos anymore. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68845 --- On Jan. 16, 2015, 12:08 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 16, 2015, 12:08 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 23, 2015, 4:25 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Changes --- This includes a rebase from the latest master. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh f692f025a7e5f2f0dddb7f6c81ea12fcb8272020 examples/vagrant/provision-dev-cluster.sh 40c165925c2110fb727c66ff5a34cf5ab8415343 examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 8428941c10641857a952f34df4e46a8fac5476a8 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py f29faf1d51baa4af66ad8c6579ffa354409e9536 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 41200bd5d74ee3239279567d79a5f48fb6af2e44 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 9fd188fb8f004d0a7664420bfda56568cebedb6f src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 7eafe074b686d55ad96018006cf4acfa823513c3 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java ad9126c32893080e128d086ea3bfd7ad23d27b89 src/test/python/apache/aurora/client/cli/test_status.py e531fa06e508d9792af51c62e67120c21baa7a81 src/test/python/apache/aurora/executor/test_thermos_executor.py 6c8ae1cf7fb578237708a954bd42162d66b39f4d Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 22, 2015, 10:42 p.m., Brian Wickman wrote:
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java,
lines 153-158
https://reviews.apache.org/r/28920/diff/18/?file=823211#file823211line153
can't the stuff in DOCKER_COMMAND_PREFIX be accomplished with a
DockerSandboxProvider in the thermos executor?
similarly, DOCKER_COMMAND_SUFFIX should be unnecessary since this
information is available in the TaskConfig.
if both the above are true, then we don't need any of the changes to
CommandUtil.
Steve Niemitz wrote:
No unfortunatly, once the executor is started it's too late to change
some things, they need to be setup before the executor launches (see my
previous comment w/ Kevin). The suffix could be moved into the executor, but
again I'd like to keep docker specific logic out of there.
Brian Wickman wrote:
it's definitely not too late. and i disagree that we should keep docker
specific logic out of the executor. that's *exactly* where it should be.
afaict, the only thing affected is LogOptions.set_log_dir('.') in
thermos_executor_main. this can easily be changed to
LogOptions.set_log_dir(os.getenv('MESOS_DIRECTORY') or '.')
*ideally* in my opinion, the logical conclusion is creating a
thermos_docker_executor_main that wires things up in accordance to the docker
philosophy, then have a completely separate docker task factory within the
scheduler to construct these. but that might be a little more unsavory in
the short term than just having the thermos_executor that understands both
containerized and non-containerized modes. there is precedence for this
already (exactly why we have sandbox providers and task providers in the
first place -- to switch implementations entirely based upon incoming
TaskConfigs.)
Steve Niemitz wrote:
I can investiate moving it into the executor, but building an entirely
seperate executor for docker is more than I'm willing to take on at this
point.
Steve Niemitz wrote:
I need to think about this more, using your suggestion of
LogOptions.set_log_dir(os.getenv('MESOS_DIRECTORY') or '.') fails because the
directory doesn't yet exist (since the symlink hasn't yet been created). Its
possible we could defer setting up logging until the executor gets a task and
set everything up in aurora_executor, but I dont know what else that might
affect.
Ok I have something working here, MESOS_DIRECTORY into the logger doesn't work
since it doesn't exist yet (and I don't want the logger to create it), but
MESOS_SANDBOX does. That plus a little extra in the DefaultSandboxProvider and
removing references to . in thermos_executor_main seems to have done the
trick. Diff coming soon.
- Steve
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/#review69288
---
On Jan. 16, 2015, 12:08 a.m., Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/
---
(Updated Jan. 16, 2015, 12:08 a.m.)
Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner.
Bugs: AURORA-633
https://issues.apache.org/jira/browse/AURORA-633
Repository: aurora
Description
---
This change adds support for launching docker containers through aurora.
These changes are based off of the discussion in
https://issues.apache.org/jira/browse/AURORA-633
As of now, a special thermos_executor.sh script is needed to launch the
executor inside docker containers. A sample aurora file is in
examples/jobs/docker.
In addition, mesos-slave must be run with `--containerizers=docker,mesos`,
the example upstart config in examples/vagrant/upstart has been updated to
reflect this.
More information is in subsequent review request comments.
Diffs
-
Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc
api/src/main/thrift/org/apache/aurora/gen/api.thrift
08ba1cdf88b712de22c26c04443079282db59ef9
config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5
docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea
docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7
examples/jobs/docker/hello_docker.aurora PRE-CREATION
examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd
examples/vagrant/provision-dev-cluster.sh
7af4b52a6876268a97630279221bb98d9b04efad
examples/vagrant/upstart/aurora-scheduler.conf
788ec254270bca074dae91829c7f4fccdc8f8bb0
examples/vagrant/upstart/mesos-slave.conf
512ce7ecf34042ed68dda55efb2dd0415f8469db
src/main/java/org/apache
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 22, 2015, 2:35 a.m., Kevin Sweeney wrote: docs/deploying-aurora-scheduler.md, line 163 https://reviews.apache.org/r/28920/diff/18/?file=823201#file823201line163 Philosophical question: if there's already a hard requirement that the container have Python 2.7 why not require that the executor be baked in as well? Maybe it's worth calling out as a TODO, but you don't have to answer it now. I think baking the executor into docker images is a recipe for disaster. Any time you upgraded aurora you'd need to then go update all containers with the new executor. Also, I don't like the idea of having to build specific aurora-isms into docker containers (I don't even really like requiring python, but that's unavoidable). On Jan. 22, 2015, 2:35 a.m., Kevin Sweeney wrote: examples/vagrant/provision-dev-cluster.sh, line 17 https://reviews.apache.org/r/28920/diff/18/?file=823204#file823204line17 nit: sh -c indirection is unnecessary here. Eh, this is just copied from the [docker install docs](https://docs.docker.com/installation/ubuntulinux/#ubuntu-precise-1204-lts-64-bit). On Jan. 22, 2015, 2:35 a.m., Kevin Sweeney wrote: src/main/python/apache/thermos/core/runner.py, lines 625-632 https://reviews.apache.org/r/28920/diff/18/?file=823218#file823218line625 Naive question: since we have this block here can we drop the preamble from the scheduler? We need both, they work together to get the logs/sandbox into the right place, and allow the observer to pick it up. On Jan. 22, 2015, 2:35 a.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, lines 147-158 https://reviews.apache.org/r/28920/diff/18/?file=823211#file823211line147 I don't see a reason the executor can't do this itself, by reading the Container field in AssignedTask and environment variables. I'd prefer not to introduce a new channel to send configuration from the scheduler to the executor with this review. @wickman might be better able to answer whether this is feasible. Some of the things above can't be (easily) done here in the executor because it's already to late by the time code in the executor begins running. For example, logging is already initialized before any real app code runs. This has gone back and forth a few times now, and we landed on keeping anything docker specific out of the executor (which I like better anyways). There's been a good amount of conversation about this in reviews above. On Jan. 22, 2015, 2:35 a.m., Kevin Sweeney wrote: src/main/python/apache/thermos/config/schema_base.py, lines 65-70 https://reviews.apache.org/r/28920/diff/18/?file=823217#file823217line65 Can we match the union-like behavior the IDL defines here, something like: ```py class Container(Struct): docker = Docker class Docker(Struct): image = Required(String) ``` This will be more easily extensible to more container types without requiring backwards-incompatible changes. I like this idea. I'll make the changes. On Jan. 22, 2015, 2:35 a.m., Kevin Sweeney wrote: docs/deploying-aurora-scheduler.md, line 158 https://reviews.apache.org/r/28920/diff/18/?file=823201#file823201line158 Latest version doesn't require a wrapper script, update docs to reflect? The previous sentence here is If using a wrapper script. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68845 --- On Jan. 16, 2015, 12:08 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 16, 2015, 12:08 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 22, 2015, 10:22 p.m., Brian Wickman wrote: src/main/python/apache/thermos/core/runner.py, lines 627-632 https://reviews.apache.org/r/28920/diff/18/?file=823218#file823218line627 this is an abstraction leak. grep the thermos codebase for 'aurora' and 'mesos'. thermos should remain aurora/mesos agnostic. thermos_runner.py includes a --sandbox flag and we should pass $MESOS_DIRECTORY to that, and not override here. the log_dir defaults to os.path.join(sandbox, '.logs'), so the log_dir part here is unnecessary. Heh, I'm pretty sure it was like that ~20 reviews ago. To be clear, your proposing passing the flag into the runner from thermos_task_runner.py? - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69282 --- On Jan. 16, 2015, 12:08 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 16, 2015, 12:08 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 7eafe074b686d55ad96018006cf4acfa823513c3 src/test
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 23, 2015, 2:43 a.m., Jay Buffington wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line 294 https://reviews.apache.org/r/28920/diff/18/?file=823211#file823211line294 Can you set force_pull_image to true here. I can't imagine why you would ever want that to be false. https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L1028 I have plans for that along with privilaged mode, but both require mesos 0.21.0 :( Once this patch is in we can start the discussion of bumping the requirement up on mesos. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69333 --- On Jan. 16, 2015, 12:08 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 16, 2015, 12:08 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 7eafe074b686d55ad96018006cf4acfa823513c3 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java ad9126c32893080e128d086ea3bfd7ad23d27b89 src
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 22, 2015, 10:42 p.m., Brian Wickman wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, lines 153-158 https://reviews.apache.org/r/28920/diff/18/?file=823211#file823211line153 can't the stuff in DOCKER_COMMAND_PREFIX be accomplished with a DockerSandboxProvider in the thermos executor? similarly, DOCKER_COMMAND_SUFFIX should be unnecessary since this information is available in the TaskConfig. if both the above are true, then we don't need any of the changes to CommandUtil. No unfortunatly, once the executor is started it's too late to change some things, they need to be setup before the executor launches (see my previous comment w/ Kevin). The suffix could be moved into the executor, but again I'd like to keep docker specific logic out of there. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69288 --- On Jan. 16, 2015, 12:08 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 16, 2015, 12:08 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 22, 2015, 10:42 p.m., Brian Wickman wrote:
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java,
lines 153-158
https://reviews.apache.org/r/28920/diff/18/?file=823211#file823211line153
can't the stuff in DOCKER_COMMAND_PREFIX be accomplished with a
DockerSandboxProvider in the thermos executor?
similarly, DOCKER_COMMAND_SUFFIX should be unnecessary since this
information is available in the TaskConfig.
if both the above are true, then we don't need any of the changes to
CommandUtil.
Steve Niemitz wrote:
No unfortunatly, once the executor is started it's too late to change
some things, they need to be setup before the executor launches (see my
previous comment w/ Kevin). The suffix could be moved into the executor, but
again I'd like to keep docker specific logic out of there.
Brian Wickman wrote:
it's definitely not too late. and i disagree that we should keep docker
specific logic out of the executor. that's *exactly* where it should be.
afaict, the only thing affected is LogOptions.set_log_dir('.') in
thermos_executor_main. this can easily be changed to
LogOptions.set_log_dir(os.getenv('MESOS_DIRECTORY') or '.')
*ideally* in my opinion, the logical conclusion is creating a
thermos_docker_executor_main that wires things up in accordance to the docker
philosophy, then have a completely separate docker task factory within the
scheduler to construct these. but that might be a little more unsavory in
the short term than just having the thermos_executor that understands both
containerized and non-containerized modes. there is precedence for this
already (exactly why we have sandbox providers and task providers in the
first place -- to switch implementations entirely based upon incoming
TaskConfigs.)
Steve Niemitz wrote:
I can investiate moving it into the executor, but building an entirely
seperate executor for docker is more than I'm willing to take on at this
point.
I need to think about this more, using your suggestion of
LogOptions.set_log_dir(os.getenv('MESOS_DIRECTORY') or '.') fails because the
directory doesn't yet exist (since the symlink hasn't yet been created). Its
possible we could defer setting up logging until the executor gets a task and
set everything up in aurora_executor, but I dont know what else that might
affect.
- Steve
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/#review69288
---
On Jan. 16, 2015, 12:08 a.m., Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/
---
(Updated Jan. 16, 2015, 12:08 a.m.)
Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner.
Bugs: AURORA-633
https://issues.apache.org/jira/browse/AURORA-633
Repository: aurora
Description
---
This change adds support for launching docker containers through aurora.
These changes are based off of the discussion in
https://issues.apache.org/jira/browse/AURORA-633
As of now, a special thermos_executor.sh script is needed to launch the
executor inside docker containers. A sample aurora file is in
examples/jobs/docker.
In addition, mesos-slave must be run with `--containerizers=docker,mesos`,
the example upstart config in examples/vagrant/upstart has been updated to
reflect this.
More information is in subsequent review request comments.
Diffs
-
Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc
api/src/main/thrift/org/apache/aurora/gen/api.thrift
08ba1cdf88b712de22c26c04443079282db59ef9
config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5
docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea
docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7
examples/jobs/docker/hello_docker.aurora PRE-CREATION
examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd
examples/vagrant/provision-dev-cluster.sh
7af4b52a6876268a97630279221bb98d9b04efad
examples/vagrant/upstart/aurora-scheduler.conf
788ec254270bca074dae91829c7f4fccdc8f8bb0
examples/vagrant/upstart/mesos-slave.conf
512ce7ecf34042ed68dda55efb2dd0415f8469db
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java
72c7545e7f16549f6a9ccb5fb74a06f154a7ea94
src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java
5226e3d1b303b1773a057078f2911c5ec2aa97f5
src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java
d885b224ec5a1d529347d84e03ba98ab6734a126
src/main/java/org/apache/aurora/scheduler/configuration
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 15, 2015, 8:24 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Changes --- Added -allowed_container_types to scheduler and decoupled Aurora config schema from Mesos for container configuration. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/configuration/ContainerType.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 7eafe074b686d55ad96018006cf4acfa823513c3 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java ad9126c32893080e128d086ea3bfd7ad23d27b89 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 16, 2015, 12:08 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 7eafe074b686d55ad96018006cf4acfa823513c3 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java ad9126c32893080e128d086ea3bfd7ad23d27b89 src/test/python/apache/aurora/client/cli/test_status.py e531fa06e508d9792af51c62e67120c21baa7a81 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 14, 2015, 7:55 p.m., Bill Farner wrote: All nits this round. Last issue for me, as you point out, is validation/restriction of `ContainerType` in `ConfigurationManager. Doing a pass on test coverage in the meantime. Bill Farner wrote: Test coverage LGTM when supplemented with https://reviews.apache.org/r/29827/ Is there any need to validate anymore now that the volume configuration is gone? That's what was going to be validated correct? - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68098 --- On Jan. 14, 2015, 7:14 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 7:14 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 8:20 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 14, 2015, 7:55 p.m., Bill Farner wrote: All nits this round. Last issue for me, as you point out, is validation/restriction of `ContainerType` in `ConfigurationManager. Doing a pass on test coverage in the meantime. Bill Farner wrote: Test coverage LGTM when supplemented with https://reviews.apache.org/r/29827/ Steve Niemitz wrote: Is there any need to validate anymore now that the volume configuration is gone? That's what was going to be validated correct? Bill Farner wrote: I would like to see an argument to specify which `ContainerType`s the scheduler should allow. This way the scheduler will not fatefully accept jobs and launch tasks that are not supported by the slave's command line and environment. Do you have any ideas on how to plumb that into the ConfigurationManager? Since it's a static class. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68098 --- On Jan. 14, 2015, 8:20 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 8:20 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 14, 2015, 2:28 a.m., Kevin Sweeney wrote: examples/jobs/docker/hello_docker.aurora, line 5 https://reviews.apache.org/r/28920/diff/13/?file=818518#file818518line5 since we're supposedly running in the python2.7 container could we show our version? Bonus points, use a python3 container to prove that this is different from the host version? If it were a python 3 container it would also need python 2.7 in order to run the executor. This limitation is called out in the docs in deploying aurora scheduler under Considerations for running jobs in docker containers however I'll add it as a note in the configuration reference as well. On Jan. 14, 2015, 2:28 a.m., Kevin Sweeney wrote: examples/vagrant/aurorabuild.sh, line 74 https://reviews.apache.org/r/28920/diff/13/?file=818519#file818519line74 is this wrapper necessary anymore? Nope, we can get rid of it. On Jan. 14, 2015, 2:28 a.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, lines 297-298 https://reviews.apache.org/r/28920/diff/13/?file=818527#file818527line297 How would you feel about removing these suffixes from the scheduler and letting the user specify an executor-per-container-type? I can see a deployment scenario where the executor is baked into the docker containers. I'll add a TODO here, but for the intial release here I don't think it's required to support that use case. On Jan. 14, 2015, 2:28 a.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line 328 https://reviews.apache.org/r/28920/diff/13/?file=818527#file818527line328 why the wrap/unwrap combination here? Thrift - Protobuf conversion. Is there a better way to do it? The ContainerInfo comment above will be the same. On Jan. 14, 2015, 2:28 a.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line 336 https://reviews.apache.org/r/28920/diff/13/?file=818527#file818527line336 Can the thermos observer detect container checkpoints any other way? IMO that would be preferable to plumbing it in on the scheduler-side. Not without big changes to the observer. This can go away if https://issues.apache.org/jira/browse/AURORA-708 is implemented. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68013 --- On Jan. 13, 2015, 1:11 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 13, 2015, 1:11 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review68062 --- Third time's the charm? I feel like I'm trying to shift without using a clutch... @ReviewBot retry - Steve Niemitz On Jan. 14, 2015, 4:50 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 14, 2015, 4:50 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 8, 2015, 6:48 p.m., Bill Farner wrote: docs/deploying-aurora-scheduler.md, line 155 https://reviews.apache.org/r/28920/diff/7/?file=808126#file808126line155 From reading the code, it appears this additional path is needed for them both to be available in the container. If so, would it be easier to just accept an arbitrary number of additional assets to copy into the sandbox? I would find that more generalized, and easier to understand. If you go with the above, i _think_ you can also safely nuke the extra args plumbing. Steve Niemitz wrote: I think we'll still need extra args (its really useful right now to pass ZK config in), but I like the idea of an arbitrary set of resources. The only trouble here would be figuring out which is the one to run and getting the symlinks right. Let's talk about this more. Bill Farner wrote: Wouldn't the extra args just be solved with the shim script? I tell the scheduler to copy `my_executor.sh` and `executor.pex` into the container, and `my_executor.sh` contains the extra args. I like this as a generalized solution to augmenting default executor behavior, since it avoids feature creep on our side just to save people the shim script. Steve Niemitz wrote: Correct, it would. My goal with this (which I'm happy to discuss more) was to make the wrapper script unnecessary in normal cases. For us we just need to pass in the ZK config for the announcer, and I feel like this is the typical case. Let me simmer on this one for a little bit. Kevin Sweeney wrote: I'd like to entirely avoid the need for a shim script by configuring the scheduler. The rationale being I would like to one day be able to host executor builds on a public artifact store (that a user doesn't necessarily have write access to) with all aurora-specific configuration happening on the scheduler. In that world the slaves would only need to know about the master, and site-specific executor settings could be configured on the scheduler like so: ``` -thermos_executor_path=https://some.public.url/thermos-0.7.0.pex -thermos_executor_flags='--announcer-enable --announcer-ensemble=zk://... ...' ``` Bill Farner wrote: The rationale being I would like to one day be able to host executor builds on a public artifact store (that a user doesn't necessarily have write access to) Is this not true today? It's not clear to me how a shim script implies write access. On the other hand, i find a host-local shim script useful for things like controlled deployment of new arguments. Bill Farner wrote: Circling back from discussion in IRC - we will proceed with two additioanl command line arguments to the scheduler: one that specifies arguments to the executor, and one that lists additional executor resources required by the executor. Please reply if you (anyone) disagree. Just to make sure we're all on the same page, does that mean running with a wrapper looks like this? -thermos_executor_path=/some/path/to/wrapper.sh -thermos_executor_resources=/some/other/path/to/thermos_executor.pex And running without looks like this: -thermos_executor_path=/some/other/path/to/thermos_executor.pex (no -thermos_executor_resources) If so I'm on board and will have a change set coming soon. One thing that comes from this however is now each sandbox will have a ~50meg pex file (the executor) copied into it. This probably isn't a huge deal, but I'd just like to point it out. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review67226 --- On Jan. 9, 2015, 7:34 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 9, 2015, 7:34 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 12, 2015, 7:01 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Changes --- Changes to thermos_executor_path / wrapper / args to reflect latest code review comments. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh b7ea41719a8f41bb23d0254e732926d89399c77c examples/vagrant/provision-dev-cluster.sh 490a8194424d4adddc2d9b3a7a7f0e579bea6c06 examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 12, 2015, 9:40 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Changes --- @ReviewBot retry Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 490a8194424d4adddc2d9b3a7a7f0e579bea6c06 examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Review Request 29827: End to end tests for docker in aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29827/ --- Review request for Aurora. Repository: aurora Description --- This adds an end to end test for docker. Diffs - examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 src/test/sh/org/apache/aurora/e2e/http/http_example_docker.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_docker_updated.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/test_common.sh 31646425233470b5f87ab50ef4504264f235f48a src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 578662ccd1735ebf500d066b3cc17b30f635c15f Diff: https://reviews.apache.org/r/29827/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29829: Fixed cleanup in end to end failure after v1 client removal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29829/ --- (Updated Jan. 12, 2015, 11:44 p.m.) Review request for Aurora and Kevin Sweeney. Repository: aurora Description --- Fixed cleanup in end to end failure after v1 client removal Diffs - src/test/sh/org/apache/aurora/e2e/test_common.sh 31646425233470b5f87ab50ef4504264f235f48a Diff: https://reviews.apache.org/r/29829/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29827: End to end tests for docker in aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29827/ --- (Updated Jan. 12, 2015, 11:44 p.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- This adds an end to end test for docker. Diffs - examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 src/test/sh/org/apache/aurora/e2e/http/http_example_docker.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_docker_updated.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/test_common.sh 31646425233470b5f87ab50ef4504264f235f48a src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 578662ccd1735ebf500d066b3cc17b30f635c15f Diff: https://reviews.apache.org/r/29827/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 8, 2015, 6:48 p.m., Bill Farner wrote: docs/deploying-aurora-scheduler.md, line 155 https://reviews.apache.org/r/28920/diff/7/?file=808126#file808126line155 From reading the code, it appears this additional path is needed for them both to be available in the container. If so, would it be easier to just accept an arbitrary number of additional assets to copy into the sandbox? I would find that more generalized, and easier to understand. If you go with the above, i _think_ you can also safely nuke the extra args plumbing. Steve Niemitz wrote: I think we'll still need extra args (its really useful right now to pass ZK config in), but I like the idea of an arbitrary set of resources. The only trouble here would be figuring out which is the one to run and getting the symlinks right. Let's talk about this more. Bill Farner wrote: Wouldn't the extra args just be solved with the shim script? I tell the scheduler to copy `my_executor.sh` and `executor.pex` into the container, and `my_executor.sh` contains the extra args. I like this as a generalized solution to augmenting default executor behavior, since it avoids feature creep on our side just to save people the shim script. Steve Niemitz wrote: Correct, it would. My goal with this (which I'm happy to discuss more) was to make the wrapper script unnecessary in normal cases. For us we just need to pass in the ZK config for the announcer, and I feel like this is the typical case. Let me simmer on this one for a little bit. Kevin Sweeney wrote: I'd like to entirely avoid the need for a shim script by configuring the scheduler. The rationale being I would like to one day be able to host executor builds on a public artifact store (that a user doesn't necessarily have write access to) with all aurora-specific configuration happening on the scheduler. In that world the slaves would only need to know about the master, and site-specific executor settings could be configured on the scheduler like so: ``` -thermos_executor_path=https://some.public.url/thermos-0.7.0.pex -thermos_executor_flags='--announcer-enable --announcer-ensemble=zk://... ...' ``` Bill Farner wrote: The rationale being I would like to one day be able to host executor builds on a public artifact store (that a user doesn't necessarily have write access to) Is this not true today? It's not clear to me how a shim script implies write access. On the other hand, i find a host-local shim script useful for things like controlled deployment of new arguments. Bill Farner wrote: Circling back from discussion in IRC - we will proceed with two additioanl command line arguments to the scheduler: one that specifies arguments to the executor, and one that lists additional executor resources required by the executor. Please reply if you (anyone) disagree. Steve Niemitz wrote: Just to make sure we're all on the same page, does that mean running with a wrapper looks like this? -thermos_executor_path=/some/path/to/wrapper.sh -thermos_executor_resources=/some/other/path/to/thermos_executor.pex And running without looks like this: -thermos_executor_path=/some/other/path/to/thermos_executor.pex (no -thermos_executor_resources) If so I'm on board and will have a change set coming soon. One thing that comes from this however is now each sandbox will have a ~50meg pex file (the executor) copied into it. This probably isn't a huge deal, but I'd just like to point it out. Bill Farner wrote: +1, and we will also have the optional argument for additional executor args. Great, this should all be incorperated into the new diffs below. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review67226 --- On Jan. 12, 2015, 9:40 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 12, 2015, 9:40 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 13, 2015, 1:11 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Changes --- Rebased Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29829: Fixed cleanup in end to end failure after v1 client removal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29829/ --- (Updated Jan. 13, 2015, 12:18 a.m.) Review request for Aurora and Kevin Sweeney. Changes --- Rebased Repository: aurora Description --- Fixed cleanup in end to end failure after v1 client removal Diffs (updated) - src/test/sh/org/apache/aurora/e2e/test_common.sh 6dc2b713f9f4f822c1d112361ff066c4abc44193 Diff: https://reviews.apache.org/r/29829/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29827: End to end tests for docker in aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29827/ --- (Updated Jan. 13, 2015, 12:32 a.m.) Review request for Aurora and Bill Farner. Changes --- Rebased from master. Repository: aurora Description --- This adds an end to end test for docker. Diffs (updated) - examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 src/test/sh/org/apache/aurora/e2e/http/http_example_docker.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_docker_updated.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 45da754341de52759d05a8960a9a978111f1e415 Diff: https://reviews.apache.org/r/29827/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29827: End to end tests for docker in aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29827/ --- (Updated Jan. 13, 2015, 1:14 a.m.) Review request for Aurora and Bill Farner. Changes --- Rebased? Repository: aurora Description --- This adds an end to end test for docker. Diffs (updated) - examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 src/test/sh/org/apache/aurora/e2e/http/http_example_docker.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_docker_updated.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 45da754341de52759d05a8960a9a978111f1e415 Diff: https://reviews.apache.org/r/29827/diff/ Testing --- Thanks, Steve Niemitz
Review Request 29829: Fixed cleanup in end to end failure after v1 client removal
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29829/ --- Review request for Aurora. Repository: aurora Description --- Fixed cleanup in end to end failure after v1 client removal Diffs - src/test/sh/org/apache/aurora/e2e/test_common.sh 31646425233470b5f87ab50ef4504264f235f48a Diff: https://reviews.apache.org/r/29829/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review67761 --- @ReviewBot retry - Steve Niemitz On Jan. 12, 2015, 9:40 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 12, 2015, 9:40 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 490a8194424d4adddc2d9b3a7a7f0e579bea6c06 examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29827: End to end tests for docker in aurora
On Jan. 12, 2015, 11:25 p.m., Kevin Sweeney wrote: src/test/sh/org/apache/aurora/e2e/test_common.sh, lines 39-40 https://reviews.apache.org/r/29827/diff/1/?file=818226#file818226line39 do you mind splitting this fix into a separate patch? If I'm reading correctly this is currently broken on master. Sure, and yeah it's broken in master as well after removing the v1 client. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29827/#review67739 --- On Jan. 12, 2015, 11:05 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29827/ --- (Updated Jan. 12, 2015, 11:05 p.m.) Review request for Aurora. Repository: aurora Description --- This adds an end to end test for docker. Diffs - examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 src/test/sh/org/apache/aurora/e2e/http/http_example_docker.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_docker_updated.aurora PRE-CREATION src/test/sh/org/apache/aurora/e2e/test_common.sh 31646425233470b5f87ab50ef4504264f235f48a src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 578662ccd1735ebf500d066b3cc17b30f635c15f Diff: https://reviews.apache.org/r/29827/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
On Jan. 8, 2015, 2:20 a.m., Kevin Sweeney wrote: Thanks for the patch, looking good, mostly style nits below. If you haven't seen them yet check out the [Java](https://github.com/twitter/commons/blob/master/src/java/com/twitter/common/styleguide.md) and [Python](https://github.com/twitter/commons/blob/master/src/python/twitter/common/styleguide.md) style guides. I think I've addressed all these, is there anything I missed? - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/#review67136 --- On Jan. 8, 2015, 5:25 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Jan. 8, 2015, 5:25 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 src/test/python/apache/aurora/executor/common/test_announcer.py e329a90b8fba43611f5120e2a5ee82220dbe2a91 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 9, 2015, 7:34 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh b7ea41719a8f41bb23d0254e732926d89399c77c examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
On Jan. 8, 2015, 2:20 a.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/app/AppModule.java, line 96 https://reviews.apache.org/r/29464/diff/8/?file=809079#file809079line96 requireNonNull I didn't add this (I just reordered the parameters), so I'm hesitant to change it. On Jan. 8, 2015, 2:20 a.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java, line 83 https://reviews.apache.org/r/29464/diff/8/?file=809080#file809080line83 im not sure why this needs to be synchronized? holding an intrinsic lock while calling out to another class is prone to deadlocks and I don't see any mutable state being protected. Good catch, I think this was an artifact of some previous refactoring. On Jan. 8, 2015, 2:20 a.m., Kevin Sweeney wrote: src/main/python/apache/aurora/executor/thermos_task_runner.py, line 76 https://reviews.apache.org/r/29464/diff/8/?file=809084#file809084line76 Can we drop the default here or make it lazy? Otherwise importing this module has the side-effect of making a syscall. That's how I had this origionally but I changed it based on [feedback above](#review66428). You're going to get a syscall regardless though. On Jan. 8, 2015, 2:20 a.m., Kevin Sweeney wrote: src/main/python/apache/aurora/executor/thermos_task_runner.py, line 101 https://reviews.apache.org/r/29464/diff/8/?file=809084#file809084line101 convention here would seem to be to make hostname default to None and do hostname or socket.gethostname() here. Same as above. I'll change it back... - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/#review67136 --- On Jan. 7, 2015, 9:21 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Jan. 7, 2015, 9:21 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 src/test/python/apache/aurora/executor/common/test_announcer.py e329a90b8fba43611f5120e2a5ee82220dbe2a91 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 8, 2015, 8:35 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh b7ea41719a8f41bb23d0254e732926d89399c77c examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 8, 2015, 6:48 p.m., Bill Farner wrote: api/src/main/thrift/org/apache/aurora/gen/api.thrift, line 207 https://reviews.apache.org/r/28920/diff/7/?file=808124#file808124line207 What are the implications of this? Does it mean a docker image must be pre-loaded on the host for it to be used? Can we cope with the user specifying a bad path? If not, what's the fallout - TASK_FAILED? Steve Niemitz wrote: The Volume options are for mounting paths on the host into the docker container, and correspond to the -v flag of docker run (https://docs.docker.com/userguide/dockervolumes/#mount-a-host-directory-as-a-data-volume). If the path is bad, it will fail but continue to run the container. This is goverened by mesos, and I actually have some plans to enhance their docker integration. Bill Farner wrote: Awesome, please include that link here and in the .md. Question remains about whether the image must be pre-loaded on the machine. Will do. I'll document it as well but the image doesn't need to be on the machine, Mesos will pull it if it doesn't exist. On Jan. 8, 2015, 6:48 p.m., Bill Farner wrote: docs/deploying-aurora-scheduler.md, line 155 https://reviews.apache.org/r/28920/diff/7/?file=808126#file808126line155 From reading the code, it appears this additional path is needed for them both to be available in the container. If so, would it be easier to just accept an arbitrary number of additional assets to copy into the sandbox? I would find that more generalized, and easier to understand. If you go with the above, i _think_ you can also safely nuke the extra args plumbing. Steve Niemitz wrote: I think we'll still need extra args (its really useful right now to pass ZK config in), but I like the idea of an arbitrary set of resources. The only trouble here would be figuring out which is the one to run and getting the symlinks right. Let's talk about this more. Bill Farner wrote: Wouldn't the extra args just be solved with the shim script? I tell the scheduler to copy `my_executor.sh` and `executor.pex` into the container, and `my_executor.sh` contains the extra args. I like this as a generalized solution to augmenting default executor behavior, since it avoids feature creep on our side just to save people the shim script. Correct, it would. My goal with this (which I'm happy to discuss more) was to make the wrapper script unnecessary in normal cases. For us we just need to pass in the ZK config for the announcer, and I feel like this is the typical case. Let me simmer on this one for a little bit. On Jan. 8, 2015, 6:48 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java, line 109 https://reviews.apache.org/r/28920/diff/7/?file=808132#file808132line109 It's good form for this type of sanitization to happen here, but at minimum the same sanitization must be done in `ConfigurationManager` to give the user a good message and avoid accepting the bad config. Steve Niemitz wrote: I do a similar check in SchedulerMain.java, ~line 211. Should I move the check into ConfigurationManager? ExecutorSettings.ctor checks as well. Bill Farner wrote: Doh, i left this comment in the wrong place. This should have been a request to sanitize the incoming thrift fields. Ah that makes more sense! I'll do so. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review67226 --- On Jan. 8, 2015, 8:35 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 8, 2015, 8:35 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 docs
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 8, 2015, 6:48 p.m., Bill Farner wrote: api/src/main/thrift/org/apache/aurora/gen/api.thrift, line 207 https://reviews.apache.org/r/28920/diff/7/?file=808124#file808124line207 What are the implications of this? Does it mean a docker image must be pre-loaded on the host for it to be used? Can we cope with the user specifying a bad path? If not, what's the fallout - TASK_FAILED? The Volume options are for mounting paths on the host into the docker container, and correspond to the -v flag of docker run (https://docs.docker.com/userguide/dockervolumes/#mount-a-host-directory-as-a-data-volume). If the path is bad, it will fail but continue to run the container. This is goverened by mesos, and I actually have some plans to enhance their docker integration. On Jan. 8, 2015, 6:48 p.m., Bill Farner wrote: api/src/main/thrift/org/apache/aurora/gen/api.thrift, line 215 https://reviews.apache.org/r/28920/diff/7/?file=808124#file808124line215 Perhaps this should be called `imageName`? Without help in the name, i could be convinced this was a path or URI. That said, what is this for? As a user, can i incorrectly specify/format this string, or is it for my own purposes? I tried to avoid a name with name in it to avoid confusion with other name fields that are purely for ID reasons (ok that sentence was a mouthful). I can enhance the docs though to be more clear that it expects a docker image name and not a URI/etc. This field is actually the docker container (image) that will be run. On Jan. 8, 2015, 6:48 p.m., Bill Farner wrote: docs/deploying-aurora-scheduler.md, line 148 https://reviews.apache.org/r/28920/diff/7/?file=808126#file808126line148 It would be really useful to see a working example of these configuration parameters in concert. I suggest you go ahead and wire it up in the upstart configs we have, and link to them from this doc. Taking it a step further, it would be awesome if this was exercised in `./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh`. Totally agree, I actually have a jira ticket (in our own jira) to do just this. I assume you mean the vagrant upstart configs? On Jan. 8, 2015, 6:48 p.m., Bill Farner wrote: docs/deploying-aurora-scheduler.md, line 153 https://reviews.apache.org/r/28920/diff/7/?file=808126#file808126line153 s/aurora/mesos/? IIUC it's the slave that does the copy. how about s/aurora will/aurora will configure mesos to/? On Jan. 8, 2015, 6:48 p.m., Bill Farner wrote: docs/deploying-aurora-scheduler.md, line 155 https://reviews.apache.org/r/28920/diff/7/?file=808126#file808126line155 From reading the code, it appears this additional path is needed for them both to be available in the container. If so, would it be easier to just accept an arbitrary number of additional assets to copy into the sandbox? I would find that more generalized, and easier to understand. If you go with the above, i _think_ you can also safely nuke the extra args plumbing. I think we'll still need extra args (its really useful right now to pass ZK config in), but I like the idea of an arbitrary set of resources. The only trouble here would be figuring out which is the one to run and getting the symlinks right. Let's talk about this more. On Jan. 8, 2015, 6:48 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java, line 108 https://reviews.apache.org/r/28920/diff/7/?file=808132#file808132line108 I applied your patch and removed these PMD exclusions without any issue. Are they needed? As a general practice - we avoid decorating the code with hints to code quality checkers. This could vary from making the code appease the checker to disabling the rule. Ah, so this used to be a != null check that PMD complained about, but since I refactored it recently to use Optional PMD stopped complaining. I'll remove these hits. On Jan. 8, 2015, 6:48 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java, line 109 https://reviews.apache.org/r/28920/diff/7/?file=808132#file808132line109 It's good form for this type of sanitization to happen here, but at minimum the same sanitization must be done in `ConfigurationManager` to give the user a good message and avoid accepting the bad config. I do a similar check in SchedulerMain.java, ~line 211. Should I move the check into ConfigurationManager? ExecutorSettings.ctor checks as well. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review67226 --- On Jan. 6, 2015, 11:32 p.m., Steve Niemitz wrote: --- This is an automatically
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Jan. 7, 2015, 9:21 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs (updated) - api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 src/test/python/apache/aurora/executor/common/test_announcer.py e329a90b8fba43611f5120e2a5ee82220dbe2a91 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Jan. 7, 2015, 4:27 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Changes --- Style chanes based on feedback. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs (updated) - api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 src/test/python/apache/aurora/executor/common/test_announcer.py e329a90b8fba43611f5120e2a5ee82220dbe2a91 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Review Request 29662: Fixed make-python-sdists
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29662/ --- Review request for Aurora. Repository: aurora Description --- Fixed make-python-sdists after removal of v1 client. Diffs - build-support/release/make-python-sdists a7a627f006b85c0749b3b7eac9528046091bf988 Diff: https://reviews.apache.org/r/29662/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29662: Fixed make-python-sdists
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29662/ --- (Updated Jan. 7, 2015, 6:18 p.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- Fixed make-python-sdists after removal of v1 client. Diffs - build-support/release/make-python-sdists a7a627f006b85c0749b3b7eac9528046091bf988 Diff: https://reviews.apache.org/r/29662/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Jan. 6, 2015, 10:45 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs (updated) - api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 src/test/python/apache/aurora/executor/common/test_announcer.py e329a90b8fba43611f5120e2a5ee82220dbe2a91 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 6, 2015, 11:32 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Changes --- Some refactoring to use Optional more. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
On Jan. 6, 2015, 9:46 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java, line 29 https://reviews.apache.org/r/29464/diff/5/?file=804823#file804823line29 There's some legacy baggage we could shed here if you're game - it should actually simplify the code and your patch if so. `LocalServiceRegistry` was designed to be an abstract registry for a framework we are moving away from, supporting an arbitrary number of services. Since we now know we _only_ have an HTTP port, we can remove the abstraction. I'm imagining this would mean extracting `JettyServerModule.HttpServerLauncher` into its own source file, and addressing the TODO to use `AbstractIdleService`. You would then add a method like `public HostAndPort getListeningAddress()`, where the host part is either the locally-resolved name or the override. That said - i understand if this yak shave exceeds your appetite, and would accept `TODO(wfarner)`. Holding on your decision here before i review other code that might change based on your path. I'd prefer to keep this patch small so it doesn't get held up a lot. However, I wouldn't mind spending some time in the near future helping out with that refactor. On Jan. 6, 2015, 9:46 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java, line 33 https://reviews.apache.org/r/29464/diff/5/?file=804823#file804823line33 Prefer OptionalString over nullable values. :thumbsup: - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/#review66920 --- On Jan. 5, 2015, 6:47 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Jan. 5, 2015, 6:47 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 src/test/python/apache/aurora/executor/common/test_announcer.py e329a90b8fba43611f5120e2a5ee82220dbe2a91 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
On Jan. 7, 2015, 1:39 a.m., Bill Farner wrote: Thanks for the patch! Most of my comments are style-related, but i think a bunch of the python changes are unneeded. Cool, I'll clean up these style comments tomorrow morning. On Jan. 7, 2015, 1:39 a.m., Bill Farner wrote: api/src/main/thrift/org/apache/thermos/thermos_internal.thrift, line 83 https://reviews.apache.org/r/29464/diff/6/?file=808041#file808041line83 AFAICT this value is not really used for anything meaningful, and no changes should be made in the runner code. I followed this value, and it is plumbed into `src/main/python/apache/thermos/observer/http/templates/rawtask.tpl`, just a title HTML tag. It's also used here I believe (which was why I had to change this part too unfortunatly) https://github.com/tellapart/incubator-aurora/blob/dns-override/src/main/python/apache/thermos/observer/http/templates/task.tpl#L60 And hostname come from here in the observer: https://github.com/tellapart/incubator-aurora/blob/dns-override/src/main/python/apache/thermos/observer/task_observer.py#L218 On Jan. 7, 2015, 1:39 a.m., Bill Farner wrote: src/main/python/apache/aurora/executor/common/announcer.py, line 79 https://reviews.apache.org/r/29464/diff/6/?file=808046#file808046line79 I assume this is effectively picking up the `--hostname` command line argument from the slave, is that correct? If so, please include a comment here about why we do that rather than calling `socket.gethostname()`. Correct, I'll add a comment to clarify. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/#review66935 --- On Jan. 6, 2015, 10:45 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Jan. 6, 2015, 10:45 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 src/test/python/apache/aurora/executor/common/test_announcer.py e329a90b8fba43611f5120e2a5ee82220dbe2a91 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 5, 2015, 8:41 p.m., Joshua Cohen wrote:
This is 99% just style nits. Unfortunately our Java styleguide isn't
published, but I'm working on rectifying that!
Thanks for all the feedback, new patch set coming up!
On Jan. 5, 2015, 8:41 p.m., Joshua Cohen wrote:
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java, lines
223-224
https://reviews.apache.org/r/28920/diff/4/?file=801606#file801606line223
Do we need to plumb the wrapper path all the way through to the
CommandUtil? What if instead of passing both along we figure out the
correct path to use here at start up and passed that along directly?
I.e. we could do something like...
OptionalString executorPath =
Optional.of(THERMOS_EXECUTOR_PATH.get()).or(Optional.of(THERMOS_EXECUTOR_WRAPPER_PATH.get()));
if (!executorPath.isPresent()) {
throw new IllegalStateException(...);
}
bind(ExecutorSettings.class).toInstance(new ExecutorSettings(
executorPath.get(),
...));
We need both paths in the ExecutorSettings so they can both get added to the
CommandInfo as resources (for docker).
On Jan. 5, 2015, 8:41 p.m., Joshua Cohen wrote:
src/test/python/apache/aurora/executor/test_thermos_executor.py, line 193
https://reviews.apache.org/r/28920/diff/4/?file=801621#file801621line193
This seems like a change in semantics... the sandbox_provider
previously was expected to be a factory function that returned the sandbox,
now you're passing in the sandbox itself? Why the change?
The UserOverrideDirectorySandboxProvider has a constructor which takes
arguments, so the provider can no longer be parameterless. I feel like this is
more true to a factory pattern anyways, instead of passing in a factory
factory we're now just passing in a factory.
- Steve
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/#review66702
---
On Jan. 5, 2015, 8:25 p.m., Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/
---
(Updated Jan. 5, 2015, 8:25 p.m.)
Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner.
Bugs: AURORA-633
https://issues.apache.org/jira/browse/AURORA-633
Repository: aurora
Description
---
This change adds support for launching docker containers through aurora.
These changes are based off of the discussion in
https://issues.apache.org/jira/browse/AURORA-633
As of now, a special thermos_executor.sh script is needed to launch the
executor inside docker containers. A sample aurora file is in
examples/jobs/docker.
In addition, mesos-slave must be run with `--containerizers=docker,mesos`,
the example upstart config in examples/vagrant/upstart has been updated to
reflect this.
More information is in subsequent review request comments.
Diffs
-
Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc
api/src/main/thrift/org/apache/aurora/gen/api.thrift
5665c69cd7b49c3fd7345074c9f16a3b224496ab
docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea
docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7
examples/jobs/docker/hello_docker.aurora PRE-CREATION
examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262
examples/vagrant/upstart/mesos-slave.conf
512ce7ecf34042ed68dda55efb2dd0415f8469db
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java
72c7545e7f16549f6a9ccb5fb74a06f154a7ea94
src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java
5226e3d1b303b1773a057078f2911c5ec2aa97f5
src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java
ead9d28100673440168a32d114ecaa15874978a6
src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java
d885b224ec5a1d529347d84e03ba98ab6734a126
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
5bf283062c9d119ff91ed45da8b236e36d0fc9aa
src/main/python/apache/aurora/config/thrift.py
ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c
src/main/python/apache/aurora/executor/aurora_executor.py
636b23d30a897b557eb8c3f8733c90b23cb807ef
src/main/python/apache/aurora/executor/bin/thermos_executor_main.py
9df9b4b79c0c7d29c5088409bf15c0d32a621df0
src/main/python/apache/aurora/executor/common/sandbox.py
f47a32b3fefb4a89940b1ddc473b8316ac00df12
src/main/python/apache/aurora/executor/thermos_task_runner.py
5e4bd65537d186459003c0b9434f1b769e04f448
src/main/python/apache/thermos/config/schema_base.py
f9143cc1b83143d6147f59d90c79435d055d0518
src/main
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 5, 2015, 11:53 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ead9d28100673440168a32d114ecaa15874978a6 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 5, 2015, 7:56 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Changes --- Updated docs and refactored CommandUtil slightly. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample script is in examples/jobs/docker, as well as an example aurora file. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. The thermos root path defaults to /var/run/thermos, however if a different path is used, it must be passed to the scheduler via `--thermos_observer_root=some path` Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ead9d28100673440168a32d114ecaa15874978a6 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 5, 2015, 8:25 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description (updated) --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ead9d28100673440168a32d114ecaa15874978a6 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Jan. 5, 2015, 6:47 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Changes --- Adding reviewers -wfarner Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 src/test/python/apache/aurora/executor/common/test_announcer.py e329a90b8fba43611f5120e2a5ee82220dbe2a91 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 5, 2015, 7:08 p.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java, line 205 https://reviews.apache.org/r/28920/diff/4/?file=801608#file801608line205 This can't be made private due to the way @Timed works (for more information see https://github.com/google/guice/wiki/AOP). Bill Farner wrote: Presumably a rebase will cause this to be caught by Maxim's new PMD rule. I think this might have been an artifact of a rebase conflict, I dont know why I would have changed this myself. Good catch though. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review66692 --- On Dec. 26, 2014, 9:05 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Dec. 26, 2014, 9:05 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample script is in examples/jobs/docker, as well as an example aurora file. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. The thermos root path defaults to /var/run/thermos, however if a different path is used, it must be passed to the scheduler via `--thermos_observer_root=some path` Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ead9d28100673440168a32d114ecaa15874978a6 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Dec. 31, 2014, 6:14 p.m.) Review request for Aurora. Changes --- This incorperates feedback from reviews. Getting the correct hostname into the checkpoint header was a little more involved than I hoped, but still cleaner than before. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs (updated) - api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 src/test/python/apache/aurora/executor/common/test_announcer.py e329a90b8fba43611f5120e2a5ee82220dbe2a91 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Dec. 31, 2014, 6:32 p.m.) Review request for Aurora. Changes --- Sorry for the spam, removed an unused import. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs (updated) - api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 src/test/python/apache/aurora/executor/common/test_announcer.py e329a90b8fba43611f5120e2a5ee82220dbe2a91 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
On Dec. 29, 2014, 8:36 p.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/configuration/LocalServiceRegistryWithOverrides.java, line 21 https://reviews.apache.org/r/29464/diff/1/?file=802787#file802787line21 Convention is to place CmdLine Args in Guice Modules (to separate wiring from implementation). Also, this value will need to be plumbed into the mesos Log somehow. Steve Niemitz wrote: Ok cool, I can make that change. Steve Niemitz wrote: Also can you explain more what you mean by plumbed into the mesos Log somehow? Kevin Sweeney wrote: When running multiple schedulers they need to communicate with each other to coordinate writes to the Mesos replicated log. They do this by advertising their IP address in zookeeper. Part of that: the scheduler IP (actually libprocess PID, which contains the scheduler IP) is written to a znode under (the value of -native_log_zk_group_path). Ah ok I understand now. The intention of this patch was more for ease of use for external clients (eg I'm not able to resolve EC2 internal IPs). Do you think this needs to change as well? I'm hesitant to change it since I haven't seen any issues with it as-is. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/#review66297 --- On Dec. 29, 2014, 8:23 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Dec. 29, 2014, 8:23 p.m.) Review request for Aurora. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -dns_name_override which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/configuration/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
On Dec. 30, 2014, 11:10 a.m., Stephan Erb wrote: src/main/java/org/apache/aurora/scheduler/configuration/LocalServiceRegistryWithOverrides.java, line 21 https://reviews.apache.org/r/29464/diff/1/?file=802787#file802787line21 To be consistent with mesos, how about calling this one --hostname? Sounds good to me, I'd been trying to think of something better than dns_name_override. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/#review66353 --- On Dec. 29, 2014, 8:23 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Dec. 29, 2014, 8:23 p.m.) Review request for Aurora. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -dns_name_override which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/configuration/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Dec. 30, 2014, 3:53 p.m.) Review request for Aurora. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -dns_name_override which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Dec. 30, 2014, 5:17 p.m.) Review request for Aurora. Repository: aurora Description (updated) --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Dec. 31, 2014, 12:23 a.m.) Review request for Aurora. Changes --- This last change also modifies thermos_executor to use the hostname specified by the mesos-slave (or the default if nothing was specified on the slave). It uses this both for publishing into ZooKeepr and writing to the checkpoint file. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/executor_base.py 1db97cc8c12752d4eca339a7680ba963a66ffbce src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
On Dec. 31, 2014, 12:51 a.m., Stephan Erb wrote: src/main/python/apache/aurora/executor/common/announcer.py, line 82 https://reviews.apache.org/r/29464/diff/3/?file=804038#file804038line82 It seems that you can simplify your code significantly if you just read the 'slaveHost' attribute form the 'AssignedTask' instance passed into your method. (I am not an Aurora committer so I be wrong here.) Stephan Erb wrote: The mentioned slaveHost seems to be extracted from mesos offers, which will be set according to your slave parameters --ip and --hostname. I'll check this out tomorrow, it'd be great if I could simplify things here and not have a dependency on the executor in the Announcer/TaskRunner. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/#review66425 --- On Dec. 31, 2014, 12:23 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Dec. 31, 2014, 12:23 a.m.) Review request for Aurora. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/executor_base.py 1db97cc8c12752d4eca339a7680ba963a66ffbce src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
On Dec. 31, 2014, 1:22 a.m., Stephan Erb wrote: src/main/python/apache/aurora/executor/thermos_task_runner.py, line 373 https://reviews.apache.org/r/29464/diff/3/?file=804040#file804040line373 How about injecting the hostname value into the constuctor of DefaultThermosTaskRunnerProvider when you create it? This would allow you to get rid of the remaining setter functions. I didn't test this, but I don't think it'll be work since the executor hasn't been initialized by mesos yet (and doesn't have the slave_info yet). If the above comment works though, this can go away as well since I can get it in from_assigned_task. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/#review66428 --- On Dec. 31, 2014, 12:23 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Dec. 31, 2014, 12:23 a.m.) Review request for Aurora. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -hostname which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - src/main/java/org/apache/aurora/scheduler/app/AppModule.java 360e161b6c3f6fd412c7e8de7f1b9a3af109593c src/main/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/announcer.py 9e5bdc3885e76d8d03aa946caac9fdec7e1e9186 src/main/python/apache/aurora/executor/executor_base.py 1db97cc8c12752d4eca339a7680ba963a66ffbce src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/test/java/org/apache/aurora/scheduler/app/LocalServiceRegistryWithOverridesTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java fbc3da3ab239b67ce3012d5a14fccd3ccb20a241 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Review Request 29464: Add option to override local scheduler address published into ZooKeeper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- Review request for Aurora. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -dns_name_override which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/configuration/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
On Dec. 29, 2014, 8:36 p.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/configuration/LocalServiceRegistryWithOverrides.java, line 39 https://reviews.apache.org/r/29464/diff/1/?file=802787#file802787line39 have you run checkstyle? method names are lowerCamelCase Yeah, good catch, I'm still not out of C# world yet. checkstyleMain didn't pick this up though oddly enough. There are a couple other things it did though, new patch set coming up with those fixes. On Dec. 29, 2014, 8:36 p.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/configuration/LocalServiceRegistryWithOverrides.java, line 51 https://reviews.apache.org/r/29464/diff/1/?file=802787#file802787line51 Why not ask for the IP address to advertise on the command-line? It looks like by the time this is called it's too late for us to do anything with a DNS resolution error. I'd rather publish the actual DNS name rather than the IP in this case (since the underlying IP could change over time). I agree though this should fail-fast at app start rather than the first time the service registry is used (which is also at startup, but it'll be more explicit the other way). On Dec. 29, 2014, 8:36 p.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/configuration/LocalServiceRegistryWithOverrides.java, line 21 https://reviews.apache.org/r/29464/diff/1/?file=802787#file802787line21 Convention is to place CmdLine Args in Guice Modules (to separate wiring from implementation). Also, this value will need to be plumbed into the mesos Log somehow. Ok cool, I can make that change. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/#review66297 --- On Dec. 29, 2014, 8:23 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Dec. 29, 2014, 8:23 p.m.) Review request for Aurora. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -dns_name_override which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/configuration/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 29464: Add option to override local scheduler address published into ZooKeeper
On Dec. 29, 2014, 8:36 p.m., Kevin Sweeney wrote: src/main/java/org/apache/aurora/scheduler/configuration/LocalServiceRegistryWithOverrides.java, line 21 https://reviews.apache.org/r/29464/diff/1/?file=802787#file802787line21 Convention is to place CmdLine Args in Guice Modules (to separate wiring from implementation). Also, this value will need to be plumbed into the mesos Log somehow. Steve Niemitz wrote: Ok cool, I can make that change. Also can you explain more what you mean by plumbed into the mesos Log somehow? - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/#review66297 --- On Dec. 29, 2014, 8:23 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29464/ --- (Updated Dec. 29, 2014, 8:23 p.m.) Review request for Aurora. Repository: aurora Description --- I've added a new flag for the aurora scheduler, -dns_name_override which can override the scheduler server address published into ZK. This is useful for cases such as running the scheduler in EC2, where the autodetected local address is actual an interal IP and not the public address of the machine. Diffs - src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/configuration/LocalServiceRegistryWithOverrides.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java cf173850635572c0df38bdd5cb14de8ce2016bf7 src/test/java/org/apache/aurora/scheduler/http/LeaderRedirectTest.java c3bf5ea4cbeaad03e187f84215b86531d55c25b3 Diff: https://reviews.apache.org/r/29464/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
On Dec. 13, 2014, 1:59 a.m., Jay Buffington wrote: I haven't had time to complete this review, but I wanted to give you what I have so far. This is all fantastic and I really appreciate you doing this! I'm excited to start using this implementation. You should update configuration-reference.md and deploying-aurora-scheduler.md in the docs dir to explain these changes. You should state minimum mesos slave version as well explain minimum docker version required on the slaves. You've made a bunch of changes to the executor and the runner to make them docker aware. I'd like those two components to not know about docker and I think we could really simplify this patch if we remove all that. I think these changes are wholly unnecessary. We should chat on IRC or Facetime about this. I don't understand how users are managed in this world. Docker images have their own /etc/passwd, so it seems to me that the unix user the mesos task runs as (same as aurora role) needs to exist inside that /etc/passwd. If that's not the case I'm confused and want to understand what user (username and id) the task does run as inside and outside the container. Allowing the aurora user to set volumes is a security nightmare. before docker we're using unix permissions to control security. We basically said: your aurora role is your unix user and that's all the permissions you get on a host. The docker daemon runs as a priviledged user, so now you can tell docker to bind mount in files from all over the system that your unix user didn't previously have permissions to read. So at a minimum, we should have a flag to disable the volumes feature with a big red warning flag in docs telling people what security issues they're signing up for whenever they enable it. Jay Buffington wrote: Oh, I forgot to mention the docs need to explain what requirements are on the docker image to run the docker executor in it. Perhaps that's just python 2.7. If that's the case, we should consider bind mounting (aka docker volumes) in (statically compiled?) py2.7 executable. Great feedback all around here, thanks for spending the time to go over this. Im on a plane into SF right now, I'll follow up with you this afternoon. On Dec. 13, 2014, 1:59 a.m., Jay Buffington wrote: examples/jobs/docker/thermos_executor.sh, line 5 https://reviews.apache.org/r/28920/diff/2/?file=789359#file789359line5 I don't think setting LIBPROCESS_IP is necessary here and I don't think you set the second arg to this anywhere in the script. Perhaps this unintentionally snuck into this patch? Good catch, that wasn't supposed to be in there. I think we can get rid of this whole example script eventually. On Dec. 13, 2014, 1:59 a.m., Jay Buffington wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line 280 https://reviews.apache.org/r/28920/diff/2/?file=789365#file789365line280 info is too verbose here. I'm just going to delete these instead, they don't add much value. On Dec. 13, 2014, 1:59 a.m., Jay Buffington wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line 308 https://reviews.apache.org/r/28920/diff/2/?file=789365#file789365line308 To make this log message useful to an operations person reading it, it should explain which job this message pertains to. As above, just removing these. On Dec. 13, 2014, 1:59 a.m., Jay Buffington wrote: src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line 285 https://reviews.apache.org/r/28920/diff/2/?file=789365#file789365line285 This is confusing to the user and I think unnecessary. It seems like the user should be able to use whatever wrapper script they want (why does it have to end in .sh?) The requirements of what the wrapper script must do should be explained in the deploying-aurora doc I actually changed this a few days ago based on internal feedback such that you pass both an executor (and or ) wrapper into the scheduler on the command line. It'll be included in this patch set. - Steve --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review65000 --- On Dec. 11, 2014, 6:16 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Dec. 11, 2014, 6:16 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Dec. 16, 2014, 9:19 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Changes --- Changes based on review feedback Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample script is in examples/jobs/docker, as well as an example aurora file. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. The thermos root path defaults to /var/run/thermos, however if a different path is used, it must be passed to the scheduler via `--thermos_observer_root=some path` Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 examples/jobs/docker/Dockerfile PRE-CREATION examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/jobs/docker/hello_docker.py PRE-CREATION examples/jobs/docker/thermos_executor.sh PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ead9d28100673440168a32d114ecaa15874978a6 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/python/apache/thermos/observer/task_observer.py cd528dcca3f5a330359cf38005f3a1a0329a4886 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- Review request for Aurora. Summary (updated) - Add support for docker containers to aurora Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description (updated) --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample script is in examples/jobs/docker, as well as an example aurora file. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. The thermos root path defaults to /var/run/thermos, however if a different path is used, it must be passed to the scheduler via `--thermos_observer_root=some path` Diffs (updated) - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 examples/jobs/docker/Dockerfile PRE-CREATION examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/jobs/docker/hello_docker.py PRE-CREATION examples/jobs/docker/thermos_executor.sh PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ead9d28100673440168a32d114ecaa15874978a6 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/python/apache/thermos/observer/task_observer.py cd528dcca3f5a330359cf38005f3a1a0329a4886 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Dec. 11, 2014, 6:16 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample script is in examples/jobs/docker, as well as an example aurora file. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. The thermos root path defaults to /var/run/thermos, however if a different path is used, it must be passed to the scheduler via `--thermos_observer_root=some path` Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 5665c69cd7b49c3fd7345074c9f16a3b224496ab api/src/main/thrift/org/apache/thermos/thermos_internal.thrift 2c449a491bc5a8ac858ea6487e4cef0591f36f66 examples/jobs/docker/Dockerfile PRE-CREATION examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/jobs/docker/hello_docker.py PRE-CREATION examples/jobs/docker/thermos_executor.sh PRE-CREATION examples/vagrant/aurorabuild.sh 69983d0140b76c6869cd04e55d760f3e3a1e4262 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ead9d28100673440168a32d114ecaa15874978a6 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/bin/thermos_runner.py 647de2771f301b17de33d8b45198c211d2e84367 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/python/apache/thermos/observer/task_observer.py cd528dcca3f5a330359cf38005f3a1a0329a4886 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 Diff: https://reviews.apache.org/r/28920/diff/ Testing --- Thanks, Steve Niemitz
