Re: Review Request 30647: Instrument the HealthChecker to export stats.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30647/#review71488
---
Master (da296a3) is red with this patch.
./build-support/jenkins/build.sh
Running setup.py install for twitter.common.collections
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.collections-0.3.0-py2.7-nspkg.pth
Running setup.py install for twitter.common.util
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.util-0.3.0-py2.7-nspkg.pth
Running setup.py install for twitter.common.log
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.log-0.3.0-py2.7-nspkg.pth
Running setup.py install for twitter.common.process
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.process-0.3.0-py2.7-nspkg.pth
Running setup.py install for gitdb
building 'gitdb._perf' extension
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2
-Wall -Wstrict-prototypes -fPIC -Igitdb -I/usr/include/python2.7 -c
gitdb/_fun.c -o build/temp.linux-x86_64-2.7/gitdb/_fun.o
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2
-Wall -Wstrict-prototypes -fPIC -Igitdb -I/usr/include/python2.7 -c
gitdb/_delta_apply.c -o build/temp.linux-x86_64-2.7/gitdb/_delta_apply.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
-Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv
-O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector
--param=ssp-buffer-size=4 -Wformat -Werror=format-security
build/temp.linux-x86_64-2.7/gitdb/_fun.o
build/temp.linux-x86_64-2.7/gitdb/_delta_apply.o -o
build/lib.linux-x86_64-2.7/gitdb/_perf.so
Running setup.py install for twitter.common.app
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.app-0.3.0-py2.7-nspkg.pth
Running setup.py install for GitPython
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/local/lib/python2.7/site-packages/setuptools/dist.py:292:
UserWarning: The version specified ('0.3.2 RC1') is an invalid version, this
may not work as expected with newer versions of setuptools, pip, and PyPI.
Please see PEP 440 for more details.
details. % self.metadata.version
Running setup.py install for pep8
Installing pep8 script to
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/bin
Running setup.py install for pyflakes
Installing pyflakes script to
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/bin
Running setup.py
Re: Review Request 30647: Instrument the HealthChecker to export stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- (Updated Feb. 6, 2015, 7:44 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Changes --- Remove no longer valid assertions about exported metrics. Bugs: AURORA-1062 https://issues.apache.org/jira/browse/AURORA-1062 Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs (updated) - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 src/test/python/apache/aurora/executor/test_thermos_executor.py c8fab307d17949a8157659c4b3944ec7520feb9d Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Review Request 30741: Add an interface and implementations of PathDetector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/ --- Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1024 https://issues.apache.org/jira/browse/AURORA-1024 Repository: aurora Description --- The goal here is that any place that uses a fixed checkpoint root that potentially needs to interact with multiple checkpoint roots (e.g. thermos observer, thermos cli, gc executor) will instead use a PathDetector implementation. The default implementation is just the FixedPathDetector to which you can pass --checkpoint_root. However in subsequent reviews, we will 1) plumb PathDetector into the abovementioned components and then 2) wire up a ChainedPathDetector that has both a FixedPathDetector and MesosPathDetector. Diffs - src/main/python/apache/aurora/executor/BUILD 79037bc5af67e5287afa2270b70992cac42de5a7 src/main/python/apache/aurora/executor/common/BUILD e64362e3b7d603f51b0545db414a3b3df414594f src/main/python/apache/aurora/executor/common/path_detector.py PRE-CREATION src/main/python/apache/aurora/executor/executor_detector.py 7b8fc4e1cf0b03f7d425a2f6bd0944b583c96737 src/main/python/apache/aurora/executor/gc_executor.py 952d77d8ff525ef069a921a987d056de81fc7476 src/main/python/apache/thermos/monitoring/BUILD 33259c854f7b28f772310620f9101fb304f30715 src/main/python/apache/thermos/monitoring/detector.py 117aef5b0f11dd33781b314be7c80cb4034cc9f8 src/test/python/apache/aurora/executor/BUILD aa3bc3b934ea40ee34241521e3f581b0d8ed0ab6 src/test/python/apache/aurora/executor/common/BUILD 9e3a657523fc9b848785fcbb0db2e737d371f08f src/test/python/apache/aurora/executor/common/test_path_detector.py PRE-CREATION src/test/python/apache/aurora/executor/test_executor_detector.py 85d541838c3f74afc022f4ea672e06b90e7e8142 src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/30741/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor/common:: src/test/python/apache/thermos/monitoring:: One failure due to incompatible merge w/ r/30704 -- will send follow up patch. Thanks, Brian Wickman
Re: Review Request 30695: Implements log rotation in the Thermos runner.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/#review71472 --- This is super rad. Thanks for taking this on. Before I do a deeper dive, what do you think about making the logrotate policy be specified by the user instead of the framework owner, with a sensible default? For example, if this is configurable on the process object, you can have different policies per process, e.g. ```py class RotatePolicy(Struct): log_size = Default(Integer, 32*MB) backups = Default(Integer, 10) copytruncate = Default(Boolean, False) compress = Default(Boolean, False) hangup_command = String ... # union class Logger(Struct): standard = Boolean # standard i/o devnull = Boolean # /dev/null redirection logrotate = RotatePolicy # use a logrotation policy DefaultLogger = Logger(standard=True) class Process(Struct): cmdline = Required(String) name= Required(String) ... logger = Default(Logger, DefaultLogger) ``` This also means reduced end-to-end plumbing through all the binaries, class constructors, etc. And if you ever need to add new features (e.g. a compress option), they're fairly well encapsulated within the Logger union. - Brian Wickman On Feb. 6, 2015, 5:51 p.m., George Sirois wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/ --- (Updated Feb. 6, 2015, 5:51 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-95 https://issues.apache.org/jira/browse/AURORA-95 Repository: aurora Description --- Implements log rotation in the Thermos runner. Diffs - docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/thermos_task_runner.py 9ff8c5379aad7ac05e44b1f5a2b74f759f26 src/main/python/apache/thermos/bin/thermos_runner.py bd8cf7f4cda54b6be72dad64f9446eedeb132211 src/main/python/apache/thermos/core/process.py 5ce138dab161d880c0bd58b87a6f5a54d4ca2f99 src/main/python/apache/thermos/core/runner.py f949f279a071c6464b026749f51afc776102f2aa src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 Diff: https://reviews.apache.org/r/30695/diff/ Testing --- ./pants test src/test/python/apache/thermos/core:all Thanks, George Sirois
Re: Review Request 30741: Add an interface and implementations of PathDetector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/#review71486 --- Ship it! Master (da296a3) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 6, 2015, 7:30 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/ --- (Updated Feb. 6, 2015, 7:30 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1024 https://issues.apache.org/jira/browse/AURORA-1024 Repository: aurora Description --- The goal here is that any place that uses a fixed checkpoint root that potentially needs to interact with multiple checkpoint roots (e.g. thermos observer, thermos cli, gc executor) will instead use a PathDetector implementation. The default implementation is just the FixedPathDetector to which you can pass --checkpoint_root. However in subsequent reviews, we will 1) plumb PathDetector into the abovementioned components and then 2) wire up a ChainedPathDetector that has both a FixedPathDetector and MesosPathDetector. Diffs - src/main/python/apache/aurora/executor/BUILD 79037bc5af67e5287afa2270b70992cac42de5a7 src/main/python/apache/aurora/executor/common/BUILD e64362e3b7d603f51b0545db414a3b3df414594f src/main/python/apache/aurora/executor/common/path_detector.py PRE-CREATION src/main/python/apache/aurora/executor/executor_detector.py 7b8fc4e1cf0b03f7d425a2f6bd0944b583c96737 src/main/python/apache/aurora/executor/gc_executor.py 952d77d8ff525ef069a921a987d056de81fc7476 src/main/python/apache/thermos/monitoring/BUILD 33259c854f7b28f772310620f9101fb304f30715 src/main/python/apache/thermos/monitoring/detector.py 117aef5b0f11dd33781b314be7c80cb4034cc9f8 src/test/python/apache/aurora/executor/BUILD aa3bc3b934ea40ee34241521e3f581b0d8ed0ab6 src/test/python/apache/aurora/executor/common/BUILD 9e3a657523fc9b848785fcbb0db2e737d371f08f src/test/python/apache/aurora/executor/common/test_path_detector.py PRE-CREATION src/test/python/apache/aurora/executor/test_executor_detector.py 85d541838c3f74afc022f4ea672e06b90e7e8142 src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/30741/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor/common:: src/test/python/apache/thermos/monitoring:: One failure due to incompatible merge w/ r/30704 -- will send follow up patch. Thanks, Brian Wickman
Re: Review Request 30741: Add an interface and implementations of PathDetector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/ --- (Updated Feb. 6, 2015, 7:30 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Fix test failure. Bugs: AURORA-1024 https://issues.apache.org/jira/browse/AURORA-1024 Repository: aurora Description --- The goal here is that any place that uses a fixed checkpoint root that potentially needs to interact with multiple checkpoint roots (e.g. thermos observer, thermos cli, gc executor) will instead use a PathDetector implementation. The default implementation is just the FixedPathDetector to which you can pass --checkpoint_root. However in subsequent reviews, we will 1) plumb PathDetector into the abovementioned components and then 2) wire up a ChainedPathDetector that has both a FixedPathDetector and MesosPathDetector. Diffs (updated) - src/main/python/apache/aurora/executor/BUILD 79037bc5af67e5287afa2270b70992cac42de5a7 src/main/python/apache/aurora/executor/common/BUILD e64362e3b7d603f51b0545db414a3b3df414594f src/main/python/apache/aurora/executor/common/path_detector.py PRE-CREATION src/main/python/apache/aurora/executor/executor_detector.py 7b8fc4e1cf0b03f7d425a2f6bd0944b583c96737 src/main/python/apache/aurora/executor/gc_executor.py 952d77d8ff525ef069a921a987d056de81fc7476 src/main/python/apache/thermos/monitoring/BUILD 33259c854f7b28f772310620f9101fb304f30715 src/main/python/apache/thermos/monitoring/detector.py 117aef5b0f11dd33781b314be7c80cb4034cc9f8 src/test/python/apache/aurora/executor/BUILD aa3bc3b934ea40ee34241521e3f581b0d8ed0ab6 src/test/python/apache/aurora/executor/common/BUILD 9e3a657523fc9b848785fcbb0db2e737d371f08f src/test/python/apache/aurora/executor/common/test_path_detector.py PRE-CREATION src/test/python/apache/aurora/executor/test_executor_detector.py 85d541838c3f74afc022f4ea672e06b90e7e8142 src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/30741/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor/common:: src/test/python/apache/thermos/monitoring:: One failure due to incompatible merge w/ r/30704 -- will send follow up patch. Thanks, Brian Wickman
Re: Review Request 30647: Instrument the HealthChecker to export stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/#review71484 --- whoops, there is a legit breakage due to the removal of the 'enabled' flags. will fix. - Brian Wickman On Feb. 5, 2015, 7:17 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- (Updated Feb. 5, 2015, 7:17 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Bugs: AURORA-1062 https://issues.apache.org/jira/browse/AURORA-1062 Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 30741: Add an interface and implementations of PathDetector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/#review71487 --- Ship it! Ship It! - Zameer Manji On Feb. 6, 2015, 11:30 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/ --- (Updated Feb. 6, 2015, 11:30 a.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1024 https://issues.apache.org/jira/browse/AURORA-1024 Repository: aurora Description --- The goal here is that any place that uses a fixed checkpoint root that potentially needs to interact with multiple checkpoint roots (e.g. thermos observer, thermos cli, gc executor) will instead use a PathDetector implementation. The default implementation is just the FixedPathDetector to which you can pass --checkpoint_root. However in subsequent reviews, we will 1) plumb PathDetector into the abovementioned components and then 2) wire up a ChainedPathDetector that has both a FixedPathDetector and MesosPathDetector. Diffs - src/main/python/apache/aurora/executor/BUILD 79037bc5af67e5287afa2270b70992cac42de5a7 src/main/python/apache/aurora/executor/common/BUILD e64362e3b7d603f51b0545db414a3b3df414594f src/main/python/apache/aurora/executor/common/path_detector.py PRE-CREATION src/main/python/apache/aurora/executor/executor_detector.py 7b8fc4e1cf0b03f7d425a2f6bd0944b583c96737 src/main/python/apache/aurora/executor/gc_executor.py 952d77d8ff525ef069a921a987d056de81fc7476 src/main/python/apache/thermos/monitoring/BUILD 33259c854f7b28f772310620f9101fb304f30715 src/main/python/apache/thermos/monitoring/detector.py 117aef5b0f11dd33781b314be7c80cb4034cc9f8 src/test/python/apache/aurora/executor/BUILD aa3bc3b934ea40ee34241521e3f581b0d8ed0ab6 src/test/python/apache/aurora/executor/common/BUILD 9e3a657523fc9b848785fcbb0db2e737d371f08f src/test/python/apache/aurora/executor/common/test_path_detector.py PRE-CREATION src/test/python/apache/aurora/executor/test_executor_detector.py 85d541838c3f74afc022f4ea672e06b90e7e8142 src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/30741/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor/common:: src/test/python/apache/thermos/monitoring:: One failure due to incompatible merge w/ r/30704 -- will send follow up patch. Thanks, Brian Wickman
Re: Review Request 30710: add mesos role feature
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/#review71407
---
Ship it!
Master (5059040) is green with this patch.
./build-support/jenkins/build.sh
I will refresh this build result if you post a review containing @ReviewBot
retry
- Aurora ReviewBot
On Feb. 6, 2015, 9:23 a.m., lozh...@ebay.com zhang wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/
---
(Updated Feb. 6, 2015, 9:23 a.m.)
Review request for Aurora and Bill Farner.
Repository: aurora
Description
---
## Problems
We are from eBay platform team. Previously, we used marathon to generate
Jenkins master instance in dedicated vms and recieve resource offer from same
dedicated vms. For the details, please refer to
http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
Now, we found Aurora is more stable and powerful. We are moving from Marathon
to Aurora. During the move, we found there is no mesos role in Aurora now.
But we need use mesos role way to solve the problem in section Frameworks
stopped receiving offers after a while of the given url.
Here is a snippet of the problem description:
*We noticed occurred after we used Marathon to create the initial set of CI
masters. As those CI masters started registering themselves as frameworks,
Marathon stopped receiving any offers from Mesos; essentially, no new CI
masters could be launched. Let’s start with Marathon. In the DRF model, it
was unfair to treat Marathon in the same bucket/role alongside hundreds of
connected Jenkins frameworks. After launching all these Jenkins frameworks,
Marathon had a large resource share and Mesos would aggressively offer
resources to frameworks that were using little or no resources. Marathon was
placed last in priority and got starved out.*
*We decided to define a dedicated Mesos role for Marathon and to have all of
the Mesos slaves that were reserved for Jenkins master instances support that
Mesos role. Jenkins frameworks were left with the default role “*”.*This
solved the problem – Mesos offered resources per role and hence Marathon
never got starved out. A framework with a special role will get resource
offers from both slaves supporting that special role and also from the
default role “*”.* However, since we were using placement constraints,
Marathon accepted resource offers only from slaves that supported both the
role and the placement constraints.*
## Solution
So we add role feature is the source code to solve the problem in same way:
When accept a resource offer, Aurora will send back the needed resources to
Mesos with the mesos role in resource offer.
How to configure the Mesos role:
1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
We change the test cases according code change. Each changed test case is
green
Merge https://github.com/zhanglong2015/incubator-aurora
Diffs
-
src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
1a158b4e0be94762ad0480e8ce74b19bacc90c97
src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java
31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58
src/main/java/org/apache/aurora/scheduler/configuration/Resources.java
b5a3140e3560f790d1db496dca3c2ee0dc96a195
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
d0994203b5650f44ca2eb32e1e2aa61875163854
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
5340d651b298ec8aa079e73d6d2f652fdf876293
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
e1c29747c9854cf75bf63f6f085cf40ca68989af
src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java
422d5a9a42310979752eb7282658316c2b772419
src/test/java/org/apache/aurora/scheduler/configuration/ResourcesTest.java
d6febb8998e05257cabe8d193cefa0b6c79f197e
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
5f08d00d39f016af9bc296e517ad49b66ab5a8de
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
411a55a8d85f60bb2703468f2d69b64b2736eee4
Diff: https://reviews.apache.org/r/30710/diff/
Testing
---
:buildSrc:compileJava UP-TO-DATE
:buildSrc:compileGroovy UP-TO-DATE
:buildSrc:processResources UP-TO-DATE
:buildSrc:classes UP-TO-DATE
:buildSrc:jar UP-TO-DATE
:buildSrc:assemble UP-TO-DATE
:buildSrc:compileTestJava UP-TO-DATE
:buildSrc:compileTestGroovy UP-TO-DATE
:buildSrc:processTestResources UP-TO-DATE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test UP-TO-DATE
:buildSrc:check UP-TO-DATE
Re: Review Request 30710: add mesos role feature
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/
---
(Updated Feb. 6, 2015, 9:23 a.m.)
Review request for Aurora and Bill Farner.
Changes
---
align code style with apache
Repository: aurora
Description
---
## Problems
We are from eBay platform team. Previously, we used marathon to generate
Jenkins master instance in dedicated vms and recieve resource offer from same
dedicated vms. For the details, please refer to
http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
Now, we found Aurora is more stable and powerful. We are moving from Marathon
to Aurora. During the move, we found there is no mesos role in Aurora now. But
we need use mesos role way to solve the problem in section Frameworks stopped
receiving offers after a while of the given url.
Here is a snippet of the problem description:
*We noticed occurred after we used Marathon to create the initial set of CI
masters. As those CI masters started registering themselves as frameworks,
Marathon stopped receiving any offers from Mesos; essentially, no new CI
masters could be launched. Let’s start with Marathon. In the DRF model, it was
unfair to treat Marathon in the same bucket/role alongside hundreds of
connected Jenkins frameworks. After launching all these Jenkins frameworks,
Marathon had a large resource share and Mesos would aggressively offer
resources to frameworks that were using little or no resources. Marathon was
placed last in priority and got starved out.*
*We decided to define a dedicated Mesos role for Marathon and to have all of
the Mesos slaves that were reserved for Jenkins master instances support that
Mesos role. Jenkins frameworks were left with the default role “*”.*This solved
the problem – Mesos offered resources per role and hence Marathon never got
starved out. A framework with a special role will get resource offers from both
slaves supporting that special role and also from the default role “*”.*
However, since we were using placement constraints, Marathon accepted resource
offers only from slaves that supported both the role and the placement
constraints.*
## Solution
So we add role feature is the source code to solve the problem in same way:
When accept a resource offer, Aurora will send back the needed resources to
Mesos with the mesos role in resource offer.
How to configure the Mesos role:
1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
We change the test cases according code change. Each changed test case is green
Merge https://github.com/zhanglong2015/incubator-aurora
Diffs (updated)
-
src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
1a158b4e0be94762ad0480e8ce74b19bacc90c97
src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java
31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58
src/main/java/org/apache/aurora/scheduler/configuration/Resources.java
b5a3140e3560f790d1db496dca3c2ee0dc96a195
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
d0994203b5650f44ca2eb32e1e2aa61875163854
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
5340d651b298ec8aa079e73d6d2f652fdf876293
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
e1c29747c9854cf75bf63f6f085cf40ca68989af
src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java
422d5a9a42310979752eb7282658316c2b772419
src/test/java/org/apache/aurora/scheduler/configuration/ResourcesTest.java
d6febb8998e05257cabe8d193cefa0b6c79f197e
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
5f08d00d39f016af9bc296e517ad49b66ab5a8de
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
411a55a8d85f60bb2703468f2d69b64b2736eee4
Diff: https://reviews.apache.org/r/30710/diff/
Testing
---
:buildSrc:compileJava UP-TO-DATE
:buildSrc:compileGroovy UP-TO-DATE
:buildSrc:processResources UP-TO-DATE
:buildSrc:classes UP-TO-DATE
:buildSrc:jar UP-TO-DATE
:buildSrc:assemble UP-TO-DATE
:buildSrc:compileTestJava UP-TO-DATE
:buildSrc:compileTestGroovy UP-TO-DATE
:buildSrc:processTestResources UP-TO-DATE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test UP-TO-DATE
:buildSrc:check UP-TO-DATE
:buildSrc:build UP-TO-DATE
:api:generateThriftJava
:api:classesThrift
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
:api:checkPython
:api:generateThriftEntitiesJava
:api:classesThriftEntities
:api:compileJava UP-TO-DATE
:api:generateThriftResources
:api:processResources UP-TO-DATE
:api:classes
:api:jar
:compileJava
Note: Writing
file:/root/incubator-aurora/dist/classes/main/com/twitter/common/args/apt/cmdline.arg.info.txt.2
:processResources
:classes
:jar
:assemble
Re: Review Request 30695: Implements log rotation in the Thermos runner.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/ --- (Updated Feb. 6, 2015, 5:14 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-95 https://issues.apache.org/jira/browse/AURORA-95 Repository: aurora Description --- Implements log rotation in the Thermos runner. Diffs (updated) - docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/main/python/apache/aurora/executor/bin/thermos_executor_main.py dd638801c377065711378b3639b98b8116d8e526 src/main/python/apache/aurora/executor/thermos_task_runner.py dd5cf12856e0412f4e2e03ad7fb5f031776c38c4 src/main/python/apache/thermos/bin/thermos_runner.py bd8cf7f4cda54b6be72dad64f9446eedeb132211 src/main/python/apache/thermos/core/process.py 5ce138dab161d880c0bd58b87a6f5a54d4ca2f99 src/main/python/apache/thermos/core/runner.py f949f279a071c6464b026749f51afc776102f2aa src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 Diff: https://reviews.apache.org/r/30695/diff/ Testing --- ./pants test src/test/python/apache/thermos/core:all Thanks, George Sirois
Re: Review Request 30695: Implements log rotation in the Thermos runner.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/#review71447 --- This patch does not apply cleanly on master (5059040), do you need to rebase? I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 6, 2015, 5:14 p.m., George Sirois wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/ --- (Updated Feb. 6, 2015, 5:14 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-95 https://issues.apache.org/jira/browse/AURORA-95 Repository: aurora Description --- Implements log rotation in the Thermos runner. Diffs - docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/main/python/apache/aurora/executor/bin/thermos_executor_main.py dd638801c377065711378b3639b98b8116d8e526 src/main/python/apache/aurora/executor/thermos_task_runner.py dd5cf12856e0412f4e2e03ad7fb5f031776c38c4 src/main/python/apache/thermos/bin/thermos_runner.py bd8cf7f4cda54b6be72dad64f9446eedeb132211 src/main/python/apache/thermos/core/process.py 5ce138dab161d880c0bd58b87a6f5a54d4ca2f99 src/main/python/apache/thermos/core/runner.py f949f279a071c6464b026749f51afc776102f2aa src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 Diff: https://reviews.apache.org/r/30695/diff/ Testing --- ./pants test src/test/python/apache/thermos/core:all Thanks, George Sirois
Re: Review Request 30650: Upgrade virtualenv to 12.0.7
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30650/#review71453 --- Ship it! Ship It! - Brian Wickman On Feb. 5, 2015, 4:13 a.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30650/ --- (Updated Feb. 5, 2015, 4:13 a.m.) Review request for Aurora, Brian Wickman and Zameer Manji. Bugs: AURORA-979 https://issues.apache.org/jira/browse/AURORA-979 Repository: aurora Description --- Upgrade virtualenv to 12.0.7 virtualenv == [12.0.7 changelog](https://pypi.python.org/pypi/virtualenv/12.0.7) and selected highlights: * Upgrade pip to 6.0.8 * Upgrade setuptools to 12.0.5 * Revert several sys.path changes new in 12.0 which were breaking virtualenv. * **PROCESS** Version numbers are now simply ``X.Y`` where the leading ``1`` has been dropped. * Now using pytest framework * Correct sys.path ordering for debian, issue #461 * Correctly throws error on older Pythons, issue #619 * Allow for empty $PATH, pull #601 * Don't set prompt if $env:VIRTUAL_ENV_DISABLE_PROMPT is set for Powershell * Updated setuptools to 7.0 Diffs - build-support/virtualenv 0175f0e38e5f995ff4e8335fb86ec14869894e69 Diff: https://reviews.apache.org/r/30650/diff/ Testing --- ```sh $ ./pants test.pytest --no-fast ./src/test/python/apache/aurora:all ``` Thanks, Joe Smith
Re: Review Request 30695: Implements log rotation in the Thermos runner.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/ --- (Updated Feb. 6, 2015, 5:51 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-95 https://issues.apache.org/jira/browse/AURORA-95 Repository: aurora Description --- Implements log rotation in the Thermos runner. Diffs (updated) - docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/thermos_task_runner.py 9ff8c5379aad7ac05e44b1f5a2b74f759f26 src/main/python/apache/thermos/bin/thermos_runner.py bd8cf7f4cda54b6be72dad64f9446eedeb132211 src/main/python/apache/thermos/core/process.py 5ce138dab161d880c0bd58b87a6f5a54d4ca2f99 src/main/python/apache/thermos/core/runner.py f949f279a071c6464b026749f51afc776102f2aa src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 Diff: https://reviews.apache.org/r/30695/diff/ Testing --- ./pants test src/test/python/apache/thermos/core:all Thanks, George Sirois
Re: Review Request 30695: Implements log rotation in the Thermos runner.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/#review71459 --- Ship it! Master (5059040) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 6, 2015, 5:51 p.m., George Sirois wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/ --- (Updated Feb. 6, 2015, 5:51 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-95 https://issues.apache.org/jira/browse/AURORA-95 Repository: aurora Description --- Implements log rotation in the Thermos runner. Diffs - docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/thermos_task_runner.py 9ff8c5379aad7ac05e44b1f5a2b74f759f26 src/main/python/apache/thermos/bin/thermos_runner.py bd8cf7f4cda54b6be72dad64f9446eedeb132211 src/main/python/apache/thermos/core/process.py 5ce138dab161d880c0bd58b87a6f5a54d4ca2f99 src/main/python/apache/thermos/core/runner.py f949f279a071c6464b026749f51afc776102f2aa src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 Diff: https://reviews.apache.org/r/30695/diff/ Testing --- ./pants test src/test/python/apache/thermos/core:all Thanks, George Sirois
