Re: Review Request 27182: Add a test for the thermos resource module
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/#review71520 --- src/test/python/apache/thermos/monitoring/test_resource.py https://reviews.apache.org/r/27182/#comment117267 these aren't explicitly covered by your BUILD target src/test/python/apache/thermos/monitoring/test_resource.py https://reviews.apache.org/r/27182/#comment117268 ditto re: coverage - Brian Wickman On Feb. 6, 2015, 10 p.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/ --- (Updated Feb. 6, 2015, 10 p.m.) Review request for Aurora and Brian Wickman. Repository: aurora Description --- Add a test for the thermos resource module Diffs - src/main/python/apache/thermos/monitoring/monitor.py 8f87f5ffc39c87e87ff78b941ea30df7138bd1ef src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_resource.py PRE-CREATION Diff: https://reviews.apache.org/r/27182/diff/ Testing --- [tw-mbp-jsmith incubator-aurora (yasumoto/thermos_test)]$ ./pants test.pytest ./src/test/python/apache/thermos/monitoring:test_resource 21:15:56 00:00 [main] (To run a reporting server: ./pants server) 21:15:56 00:00 [bootstrap] 21:15:57 00:01 [setup] 21:15:57 00:01 [parse] Executing tasks in goals: bootstrap - imports - gen - check-exclusives - resolve - compile - resources - test 21:15:57 00:01 [bootstrap] 21:15:57 00:01 [bootstrap-jvm-tools] 21:15:57 00:01 [imports] 21:15:57 00:01 [ivy-imports] 21:15:57 00:01 [gen] 21:15:57 00:01 [thrift] 21:15:57 00:01 [scrooge] 21:15:57 00:01 [protoc] 21:15:57 00:01 [antlr] 21:15:57 00:01 [ragel] 21:15:57 00:01 [jaxb] 21:15:57 00:01 [wire] 21:15:57 00:01 [aapt] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [resolve] 21:15:57 00:01 [ivy] 21:15:57 00:01 [compile] 21:15:57 00:01 [jvm] 21:15:57 00:01 [jvm-compilers] 21:15:57 00:01 [resources] 21:15:57 00:01 [prepare] 21:15:57 00:01 [test] 21:15:57 00:01 [run_prep_command] 21:15:57 00:01 [pytest] 21:15:57 00:01 [run] == test session starts === platform darwin -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4 plugins: cov, timeout collected 5 items src/test/python/apache/thermos/monitoring/test_resource.py . 5 passed in 0.18 seconds 21:16:04 00:08 [junit] 21:16:04 00:08 [specs] SUCCESS Thanks, Joe Smith
Review Request 30768: Reject None values for TaskPath
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/ --- Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-1115 https://issues.apache.org/jira/browse/AURORA-1115 Repository: aurora Description --- This patch modifies TaskPath to reject None values. Diffs - src/main/python/apache/aurora/executor/thermos_task_runner.py 7b346e253677ee9b42c57782f7f67ff63b6a0083 src/main/python/apache/thermos/common/path.py 9e617051f16f4270b3958f48e0cc43706d245eec src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py PRE-CREATION src/test/python/apache/thermos/common/test_pathspec.py 3437b196d33d7c2ff6ba292ff99b6881954e7ecb Diff: https://reviews.apache.org/r/30768/diff/ Testing --- ./pants test src/test/python/apache/thermos:: ./pants test src/test/python/apache/aurora/executor:: ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Zameer Manji
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71567 --- pants.ini https://reviews.apache.org/r/30749/#comment117327 This not does not appear to be a change related to your work here. src/main/python/apache/aurora/executor/gc_executor.py https://reviews.apache.org/r/30749/#comment117328 What's the benefit of using a namedtuple here? src/main/python/apache/aurora/executor/gc_executor.py https://reviews.apache.org/r/30749/#comment117329 Shouldn't this log messsage be inside the collector implementation? - Zameer Manji On Feb. 6, 2015, 6:27 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 6, 2015, 6:27 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - pants.ini 99648e46ca755c3c8e22d90c7b682107d8dee333 src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 27182: Add a test for the thermos resource module
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/#review71523 --- Tried to merge but got a conflict. Could you re-merge this with master and re-publish so that I can patch to master? - Brian Wickman On Feb. 6, 2015, 10 p.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/ --- (Updated Feb. 6, 2015, 10 p.m.) Review request for Aurora and Brian Wickman. Repository: aurora Description --- Add a test for the thermos resource module Diffs - src/main/python/apache/thermos/monitoring/monitor.py 8f87f5ffc39c87e87ff78b941ea30df7138bd1ef src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_resource.py PRE-CREATION Diff: https://reviews.apache.org/r/27182/diff/ Testing --- [tw-mbp-jsmith incubator-aurora (yasumoto/thermos_test)]$ ./pants test.pytest ./src/test/python/apache/thermos/monitoring:test_resource 21:15:56 00:00 [main] (To run a reporting server: ./pants server) 21:15:56 00:00 [bootstrap] 21:15:57 00:01 [setup] 21:15:57 00:01 [parse] Executing tasks in goals: bootstrap - imports - gen - check-exclusives - resolve - compile - resources - test 21:15:57 00:01 [bootstrap] 21:15:57 00:01 [bootstrap-jvm-tools] 21:15:57 00:01 [imports] 21:15:57 00:01 [ivy-imports] 21:15:57 00:01 [gen] 21:15:57 00:01 [thrift] 21:15:57 00:01 [scrooge] 21:15:57 00:01 [protoc] 21:15:57 00:01 [antlr] 21:15:57 00:01 [ragel] 21:15:57 00:01 [jaxb] 21:15:57 00:01 [wire] 21:15:57 00:01 [aapt] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [resolve] 21:15:57 00:01 [ivy] 21:15:57 00:01 [compile] 21:15:57 00:01 [jvm] 21:15:57 00:01 [jvm-compilers] 21:15:57 00:01 [resources] 21:15:57 00:01 [prepare] 21:15:57 00:01 [test] 21:15:57 00:01 [run_prep_command] 21:15:57 00:01 [pytest] 21:15:57 00:01 [run] == test session starts === platform darwin -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4 plugins: cov, timeout collected 5 items src/test/python/apache/thermos/monitoring/test_resource.py . 5 passed in 0.18 seconds 21:16:04 00:08 [junit] 21:16:04 00:08 [specs] SUCCESS Thanks, Joe Smith
Review Request 30752: Fix executor builds.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30752/ --- Review request for Aurora. Repository: aurora Description --- The executor binary targets were broken when I moved the executor_detector from apache.aurora.executor to apache.aurora.executor.common Diffs - src/main/python/apache/aurora/executor/bin/BUILD 4d4f3ab9442ceda912ac4ce07eb15e291a4ed08c Diff: https://reviews.apache.org/r/30752/diff/ Testing --- Built the binaries. Thanks, Brian Wickman
Re: Review Request 27182: Add a test for the thermos resource module
On Feb. 6, 2015, 3:11 p.m., Brian Wickman wrote: In practice I've found that this helps catch issues sooner than later, as we'll see missing dependencies from the main target (which I've done before). Happy to move away if you disagree though (I know it's a tradeoff) - Joe --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/#review71520 --- On Feb. 6, 2015, 2 p.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/ --- (Updated Feb. 6, 2015, 2 p.m.) Review request for Aurora and Brian Wickman. Repository: aurora Description --- Add a test for the thermos resource module Diffs - src/main/python/apache/thermos/monitoring/monitor.py 8f87f5ffc39c87e87ff78b941ea30df7138bd1ef src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_resource.py PRE-CREATION Diff: https://reviews.apache.org/r/27182/diff/ Testing --- [tw-mbp-jsmith incubator-aurora (yasumoto/thermos_test)]$ ./pants test.pytest ./src/test/python/apache/thermos/monitoring:test_resource 21:15:56 00:00 [main] (To run a reporting server: ./pants server) 21:15:56 00:00 [bootstrap] 21:15:57 00:01 [setup] 21:15:57 00:01 [parse] Executing tasks in goals: bootstrap - imports - gen - check-exclusives - resolve - compile - resources - test 21:15:57 00:01 [bootstrap] 21:15:57 00:01 [bootstrap-jvm-tools] 21:15:57 00:01 [imports] 21:15:57 00:01 [ivy-imports] 21:15:57 00:01 [gen] 21:15:57 00:01 [thrift] 21:15:57 00:01 [scrooge] 21:15:57 00:01 [protoc] 21:15:57 00:01 [antlr] 21:15:57 00:01 [ragel] 21:15:57 00:01 [jaxb] 21:15:57 00:01 [wire] 21:15:57 00:01 [aapt] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [resolve] 21:15:57 00:01 [ivy] 21:15:57 00:01 [compile] 21:15:57 00:01 [jvm] 21:15:57 00:01 [jvm-compilers] 21:15:57 00:01 [resources] 21:15:57 00:01 [prepare] 21:15:57 00:01 [test] 21:15:57 00:01 [run_prep_command] 21:15:57 00:01 [pytest] 21:15:57 00:01 [run] == test session starts === platform darwin -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4 plugins: cov, timeout collected 5 items src/test/python/apache/thermos/monitoring/test_resource.py . 5 passed in 0.18 seconds 21:16:04 00:08 [junit] 21:16:04 00:08 [specs] SUCCESS Thanks, Joe Smith
Re: Review Request 27182: Add a test for the thermos resource module
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/#review71522 --- Ship it! Ship It! - Brian Wickman On Feb. 6, 2015, 10 p.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/ --- (Updated Feb. 6, 2015, 10 p.m.) Review request for Aurora and Brian Wickman. Repository: aurora Description --- Add a test for the thermos resource module Diffs - src/main/python/apache/thermos/monitoring/monitor.py 8f87f5ffc39c87e87ff78b941ea30df7138bd1ef src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_resource.py PRE-CREATION Diff: https://reviews.apache.org/r/27182/diff/ Testing --- [tw-mbp-jsmith incubator-aurora (yasumoto/thermos_test)]$ ./pants test.pytest ./src/test/python/apache/thermos/monitoring:test_resource 21:15:56 00:00 [main] (To run a reporting server: ./pants server) 21:15:56 00:00 [bootstrap] 21:15:57 00:01 [setup] 21:15:57 00:01 [parse] Executing tasks in goals: bootstrap - imports - gen - check-exclusives - resolve - compile - resources - test 21:15:57 00:01 [bootstrap] 21:15:57 00:01 [bootstrap-jvm-tools] 21:15:57 00:01 [imports] 21:15:57 00:01 [ivy-imports] 21:15:57 00:01 [gen] 21:15:57 00:01 [thrift] 21:15:57 00:01 [scrooge] 21:15:57 00:01 [protoc] 21:15:57 00:01 [antlr] 21:15:57 00:01 [ragel] 21:15:57 00:01 [jaxb] 21:15:57 00:01 [wire] 21:15:57 00:01 [aapt] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [resolve] 21:15:57 00:01 [ivy] 21:15:57 00:01 [compile] 21:15:57 00:01 [jvm] 21:15:57 00:01 [jvm-compilers] 21:15:57 00:01 [resources] 21:15:57 00:01 [prepare] 21:15:57 00:01 [test] 21:15:57 00:01 [run_prep_command] 21:15:57 00:01 [pytest] 21:15:57 00:01 [run] == test session starts === platform darwin -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4 plugins: cov, timeout collected 5 items src/test/python/apache/thermos/monitoring/test_resource.py . 5 passed in 0.18 seconds 21:16:04 00:08 [junit] 21:16:04 00:08 [specs] SUCCESS Thanks, Joe Smith
Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 4d4f3ab9442ceda912ac4ce07eb15e291a4ed08c src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30749/#review71538
---
Master (11a65d2) is red with this patch.
./build-support/jenkins/build.sh
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.util-0.3.0-py2.7-nspkg.pth
Running setup.py install for twitter.common.log
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.log-0.3.0-py2.7-nspkg.pth
Running setup.py install for twitter.common.process
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.process-0.3.0-py2.7-nspkg.pth
Running setup.py install for gitdb
building 'gitdb._perf' extension
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2
-Wall -Wstrict-prototypes -fPIC -Igitdb -I/usr/include/python2.7 -c
gitdb/_fun.c -o build/temp.linux-x86_64-2.7/gitdb/_fun.o
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2
-Wall -Wstrict-prototypes -fPIC -Igitdb -I/usr/include/python2.7 -c
gitdb/_delta_apply.c -o build/temp.linux-x86_64-2.7/gitdb/_delta_apply.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
-Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv
-O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector
--param=ssp-buffer-size=4 -Wformat -Werror=format-security
build/temp.linux-x86_64-2.7/gitdb/_fun.o
build/temp.linux-x86_64-2.7/gitdb/_delta_apply.o -o
build/lib.linux-x86_64-2.7/gitdb/_perf.so
Running setup.py install for twitter.common.app
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.app-0.3.0-py2.7-nspkg.pth
Running setup.py install for GitPython
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/local/lib/python2.7/site-packages/setuptools/dist.py:292:
UserWarning: The version specified ('0.3.2 RC1') is an invalid version, this
may not work as expected with newer versions of setuptools, pip, and PyPI.
Please see PEP 440 for more details.
details. % self.metadata.version
Running setup.py install for pep8
Installing pep8 script to
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/bin
Running setup.py install for pyflakes
Installing pyflakes script to
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/bin
Running setup.py install for twitter.checkstyle
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.checkstyle-0.1.0-py2.7-nspkg.pth
Installing twitterstyle script to
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/bin
Successfully installed GitPython-0.3.2rc1 gitdb-0.6.4 pep8-1.4.5 pyflakes-0.7.2
smmap-0.9.0 twitter.checkstyle-0.1.0 twitter.common.app-0.3.0
twitter.common.collections-0.3.0 twitter.common.contextutil-0.3.0
twitter.common.dirutil-0.3.0 twitter.common.lang-0.3.0 twitter.common.log-0.3.0
twitter.common.options-0.3.0
Re: Review Request 27182: Add a test for the thermos resource module
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/ --- (Updated Feb. 6, 2015, 3:55 p.m.) Review request for Aurora and Brian Wickman. Repository: aurora Description --- Add a test for the thermos resource module Diffs (updated) - src/main/python/apache/thermos/monitoring/monitor.py 8f87f5ffc39c87e87ff78b941ea30df7138bd1ef src/test/python/apache/thermos/monitoring/BUILD 1b829156900711d9f21b940c024adff664c5c5f5 src/test/python/apache/thermos/monitoring/test_resource.py PRE-CREATION Diff: https://reviews.apache.org/r/27182/diff/ Testing --- [tw-mbp-jsmith incubator-aurora (yasumoto/thermos_test)]$ ./pants test.pytest ./src/test/python/apache/thermos/monitoring:test_resource 21:15:56 00:00 [main] (To run a reporting server: ./pants server) 21:15:56 00:00 [bootstrap] 21:15:57 00:01 [setup] 21:15:57 00:01 [parse] Executing tasks in goals: bootstrap - imports - gen - check-exclusives - resolve - compile - resources - test 21:15:57 00:01 [bootstrap] 21:15:57 00:01 [bootstrap-jvm-tools] 21:15:57 00:01 [imports] 21:15:57 00:01 [ivy-imports] 21:15:57 00:01 [gen] 21:15:57 00:01 [thrift] 21:15:57 00:01 [scrooge] 21:15:57 00:01 [protoc] 21:15:57 00:01 [antlr] 21:15:57 00:01 [ragel] 21:15:57 00:01 [jaxb] 21:15:57 00:01 [wire] 21:15:57 00:01 [aapt] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [resolve] 21:15:57 00:01 [ivy] 21:15:57 00:01 [compile] 21:15:57 00:01 [jvm] 21:15:57 00:01 [jvm-compilers] 21:15:57 00:01 [resources] 21:15:57 00:01 [prepare] 21:15:57 00:01 [test] 21:15:57 00:01 [run_prep_command] 21:15:57 00:01 [pytest] 21:15:57 00:01 [run] == test session starts === platform darwin -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4 plugins: cov, timeout collected 5 items src/test/python/apache/thermos/monitoring/test_resource.py . 5 passed in 0.18 seconds 21:16:04 00:08 [junit] 21:16:04 00:08 [specs] SUCCESS Thanks, Joe Smith
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 7, 2015, 2:27 a.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Fix checkstyle + legit flakes bug! Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - pants.ini 99648e46ca755c3c8e22d90c7b682107d8dee333 src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30647: Instrument the HealthChecker to export stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- (Updated Feb. 6, 2015, 11:13 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Changes --- Fix style violation. Bugs: AURORA-1062 https://issues.apache.org/jira/browse/AURORA-1062 Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs (updated) - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 src/test/python/apache/aurora/executor/test_thermos_executor.py c8fab307d17949a8157659c4b3944ec7520feb9d Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 30225: Modifying update controller to support heartbeats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30225/#review71545 --- Ship it! Master (11a65d2) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 7, 2015, 2:43 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30225/ --- (Updated Feb. 7, 2015, 2:43 a.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, and Bill Farner. Bugs: AURORA-1010 https://issues.apache.org/jira/browse/AURORA-1010 Repository: aurora Description --- Added pulsing support into the JobUpdateController. The qualified coordinated updates get blocked until a pulse arrives. An update then becomes active and proceeds until `blockIfNoPulsesAfterMs` expires or the update reaches a terminal state (whichever comes first). UPDATE: - Added explicit states to capture blocked updates - Refactored pulseUpdate() to not rely on DB state - Dropped JobUpdatePulseState.PAUSED as it does not seem necessary. Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift 4f603f9e7ed004e53937a45ea8edf7241e15f5cf src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java 7f2ec71d744d5fafac84291cc79feba3398d0e1e src/main/java/org/apache/aurora/scheduler/storage/JobUpdateStore.java b7d8d524e15f101416889c00efc3ecf2c8d9c1a1 src/main/java/org/apache/aurora/scheduler/storage/db/DBJobUpdateStore.java d479d20259f284528b2291e2e486b36d8e47fe5e src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java 60f535988a20638fb16515d5027bfa65f1afb73c src/main/java/org/apache/aurora/scheduler/updater/JobUpdateController.java d3b30d48b76d8d7c64cda006a34f7ed3296526f2 src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java ec9d1e07abca1bf30262e1c0f741a34741e96f5d src/main/java/org/apache/aurora/scheduler/updater/JobUpdateStateMachine.java 76460f95e058181b711fb6869f5a34c1d5bdfe31 src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml f9c9ceddc559b43b4a5c45c745d54ff47484edde src/test/java/org/apache/aurora/scheduler/storage/db/DBJobUpdateStoreTest.java ca7c0c2675477cc727ca006697665f997972dfde src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 92cfa2b30c1c4daa3ae225fc8609fbeaecdaff7a src/test/java/org/apache/aurora/scheduler/updater/JobUpdateStateMachineTest.java 89765ac3d34a827d3748fb96a275d78e9d1b8b72 src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 41d422939209d0808235128e4242c11e8ef25969 Diff: https://reviews.apache.org/r/30225/diff/ Testing --- ./gradlew -Pq build Thanks, Maxim Khutornenko
Re: Review Request 30768: Reject None values for TaskPath
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/#review71568 --- Ship it! Master (11a65d2) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 8, 2015, 2:06 a.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/ --- (Updated Feb. 8, 2015, 2:06 a.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-1115 https://issues.apache.org/jira/browse/AURORA-1115 Repository: aurora Description --- This patch modifies TaskPath to reject None values. Diffs - src/main/python/apache/aurora/executor/thermos_task_runner.py 7b346e253677ee9b42c57782f7f67ff63b6a0083 src/main/python/apache/thermos/common/path.py 9e617051f16f4270b3958f48e0cc43706d245eec src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py PRE-CREATION src/test/python/apache/thermos/common/test_pathspec.py 3437b196d33d7c2ff6ba292ff99b6881954e7ecb Diff: https://reviews.apache.org/r/30768/diff/ Testing --- ./pants test src/test/python/apache/thermos:: ./pants test src/test/python/apache/aurora/executor:: ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Zameer Manji
Re: Review Request 30752: Fix executor builds.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30752/#review71527 --- Ship it! Ship It! - Joe Smith On Feb. 6, 2015, 3:52 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30752/ --- (Updated Feb. 6, 2015, 3:52 p.m.) Review request for Aurora and Joe Smith. Repository: aurora Description --- The executor binary targets were broken when I moved the executor_detector from apache.aurora.executor to apache.aurora.executor.common Diffs - src/main/python/apache/aurora/executor/bin/BUILD 4d4f3ab9442ceda912ac4ce07eb15e291a4ed08c Diff: https://reviews.apache.org/r/30752/diff/ Testing --- Built the binaries. Thanks, Brian Wickman
Re: Review Request 27182: Add a test for the thermos resource module
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/#review71510 --- Out of curiosity, why this test now? I sort of see this code on its deathbed as soon Mesos will have disk enforcement built in. - Brian Wickman On Feb. 6, 2015, 7:59 p.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/ --- (Updated Feb. 6, 2015, 7:59 p.m.) Review request for Aurora, Kevin Sweeney and Brian Wickman. Repository: aurora Description --- Add a test for the thermos resource module Diffs - src/main/python/apache/thermos/monitoring/monitor.py 8f87f5ffc39c87e87ff78b941ea30df7138bd1ef src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_resource.py PRE-CREATION Diff: https://reviews.apache.org/r/27182/diff/ Testing --- [tw-mbp-jsmith incubator-aurora (yasumoto/thermos_test)]$ ./pants test.pytest ./src/test/python/apache/thermos/monitoring:test_resource 21:15:56 00:00 [main] (To run a reporting server: ./pants server) 21:15:56 00:00 [bootstrap] 21:15:57 00:01 [setup] 21:15:57 00:01 [parse] Executing tasks in goals: bootstrap - imports - gen - check-exclusives - resolve - compile - resources - test 21:15:57 00:01 [bootstrap] 21:15:57 00:01 [bootstrap-jvm-tools] 21:15:57 00:01 [imports] 21:15:57 00:01 [ivy-imports] 21:15:57 00:01 [gen] 21:15:57 00:01 [thrift] 21:15:57 00:01 [scrooge] 21:15:57 00:01 [protoc] 21:15:57 00:01 [antlr] 21:15:57 00:01 [ragel] 21:15:57 00:01 [jaxb] 21:15:57 00:01 [wire] 21:15:57 00:01 [aapt] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [resolve] 21:15:57 00:01 [ivy] 21:15:57 00:01 [compile] 21:15:57 00:01 [jvm] 21:15:57 00:01 [jvm-compilers] 21:15:57 00:01 [resources] 21:15:57 00:01 [prepare] 21:15:57 00:01 [test] 21:15:57 00:01 [run_prep_command] 21:15:57 00:01 [pytest] 21:15:57 00:01 [run] == test session starts === platform darwin -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4 plugins: cov, timeout collected 5 items src/test/python/apache/thermos/monitoring/test_resource.py . 5 passed in 0.18 seconds 21:16:04 00:08 [junit] 21:16:04 00:08 [specs] SUCCESS Thanks, Joe Smith
Re: Review Request 30741: Add an interface and implementations of PathDetector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/#review71505 --- Ship it! These are all nits, looks good to me. src/main/python/apache/thermos/monitoring/detector.py https://reviews.apache.org/r/30741/#comment117255 Just for my own education, what's the difference between returning `self._paths` and `self._paths[:]`? src/test/python/apache/aurora/executor/common/test_path_detector.py https://reviews.apache.org/r/30741/#comment117256 nit: pull '/var/blah/blah' out to a var and use it in the assert below? Same for 'ckpt' src/test/python/apache/thermos/monitoring/test_detector.py https://reviews.apache.org/r/30741/#comment117258 same here, extract a var and resuse for the call and the assert? (same for the test below as well) src/test/python/apache/thermos/monitoring/test_detector.py https://reviews.apache.org/r/30741/#comment117259 pull this out to a separate test? (also, same for the test below, could be 3 separate tests) - Joshua Cohen On Feb. 6, 2015, 7:30 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/ --- (Updated Feb. 6, 2015, 7:30 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1024 https://issues.apache.org/jira/browse/AURORA-1024 Repository: aurora Description --- The goal here is that any place that uses a fixed checkpoint root that potentially needs to interact with multiple checkpoint roots (e.g. thermos observer, thermos cli, gc executor) will instead use a PathDetector implementation. The default implementation is just the FixedPathDetector to which you can pass --checkpoint_root. However in subsequent reviews, we will 1) plumb PathDetector into the abovementioned components and then 2) wire up a ChainedPathDetector that has both a FixedPathDetector and MesosPathDetector. Diffs - src/main/python/apache/aurora/executor/BUILD 79037bc5af67e5287afa2270b70992cac42de5a7 src/main/python/apache/aurora/executor/common/BUILD e64362e3b7d603f51b0545db414a3b3df414594f src/main/python/apache/aurora/executor/common/path_detector.py PRE-CREATION src/main/python/apache/aurora/executor/executor_detector.py 7b8fc4e1cf0b03f7d425a2f6bd0944b583c96737 src/main/python/apache/aurora/executor/gc_executor.py 952d77d8ff525ef069a921a987d056de81fc7476 src/main/python/apache/thermos/monitoring/BUILD 33259c854f7b28f772310620f9101fb304f30715 src/main/python/apache/thermos/monitoring/detector.py 117aef5b0f11dd33781b314be7c80cb4034cc9f8 src/test/python/apache/aurora/executor/BUILD aa3bc3b934ea40ee34241521e3f581b0d8ed0ab6 src/test/python/apache/aurora/executor/common/BUILD 9e3a657523fc9b848785fcbb0db2e737d371f08f src/test/python/apache/aurora/executor/common/test_path_detector.py PRE-CREATION src/test/python/apache/aurora/executor/test_executor_detector.py 85d541838c3f74afc022f4ea672e06b90e7e8142 src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/30741/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor/common:: src/test/python/apache/thermos/monitoring:: One failure due to incompatible merge w/ r/30704 -- will send follow up patch. Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 6, 2015, 11:41 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Update updates[] to only map task_ids. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - examples/jobs/cron_hello_world.aurora a99b3ab7b1c82e4e8d72ac7dece6e44d1bba498f src/main/python/apache/aurora/executor/bin/BUILD 4d4f3ab9442ceda912ac4ce07eb15e291a4ed08c src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30695: Implements log rotation in the Thermos runner.
On Feb. 6, 2015, 6:52 p.m., Brian Wickman wrote: This is super rad. Thanks for taking this on. Before I do a deeper dive, what do you think about making the logrotate policy be specified by the user instead of the framework owner, with a sensible default? For example, if this is configurable on the process object, you can have different policies per process, e.g. ```py class RotatePolicy(Struct): log_size = Default(Integer, 32*MB) backups = Default(Integer, 10) copytruncate = Default(Boolean, False) compress = Default(Boolean, False) hangup_command = String ... # union class Logger(Struct): standard = Boolean # standard i/o devnull = Boolean # /dev/null redirection logrotate = RotatePolicy # use a logrotation policy DefaultLogger = Logger(standard=True) class Process(Struct): cmdline = Required(String) name= Required(String) ... logger = Default(Logger, DefaultLogger) ``` This also means reduced end-to-end plumbing through all the binaries, class constructors, etc. And if you ever need to add new features (e.g. a compress option), they're fairly well encapsulated within the Logger union. Awesome, thanks for the feedback. I'd be willing to take this on; it would definitely make the plumbing a lot cleaner and provide more flexibility, although the downside is that it's now harder to apply a universal default (besides whatever we arrive at as the Aurora default). I'll be able to pick this up next week and can probably have a modified review out by Wednesday evening. What do you think about starting out with a simple configuration (just log_size and backups on RotatePolicy) and iterating from there? I also have one question - what distinction are you making between the standard flag on Logger and the existence of a rotation policy? - George --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/#review71472 --- On Feb. 6, 2015, 5:51 p.m., George Sirois wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/ --- (Updated Feb. 6, 2015, 5:51 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-95 https://issues.apache.org/jira/browse/AURORA-95 Repository: aurora Description --- Implements log rotation in the Thermos runner. Diffs - docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/thermos_task_runner.py 9ff8c5379aad7ac05e44b1f5a2b74f759f26 src/main/python/apache/thermos/bin/thermos_runner.py bd8cf7f4cda54b6be72dad64f9446eedeb132211 src/main/python/apache/thermos/core/process.py 5ce138dab161d880c0bd58b87a6f5a54d4ca2f99 src/main/python/apache/thermos/core/runner.py f949f279a071c6464b026749f51afc776102f2aa src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 Diff: https://reviews.apache.org/r/30695/diff/ Testing --- ./pants test src/test/python/apache/thermos/core:all Thanks, George Sirois
Re: Review Request 30695: Implements log rotation in the Thermos runner.
On Feb. 6, 2015, 6:52 p.m., Brian Wickman wrote: This is super rad. Thanks for taking this on. Before I do a deeper dive, what do you think about making the logrotate policy be specified by the user instead of the framework owner, with a sensible default? For example, if this is configurable on the process object, you can have different policies per process, e.g. ```py class RotatePolicy(Struct): log_size = Default(Integer, 32*MB) backups = Default(Integer, 10) copytruncate = Default(Boolean, False) compress = Default(Boolean, False) hangup_command = String ... # union class Logger(Struct): standard = Boolean # standard i/o devnull = Boolean # /dev/null redirection logrotate = RotatePolicy # use a logrotation policy DefaultLogger = Logger(standard=True) class Process(Struct): cmdline = Required(String) name= Required(String) ... logger = Default(Logger, DefaultLogger) ``` This also means reduced end-to-end plumbing through all the binaries, class constructors, etc. And if you ever need to add new features (e.g. a compress option), they're fairly well encapsulated within the Logger union. George Sirois wrote: Awesome, thanks for the feedback. I'd be willing to take this on; it would definitely make the plumbing a lot cleaner and provide more flexibility, although the downside is that it's now harder to apply a universal default (besides whatever we arrive at as the Aurora default). I'll be able to pick this up next week and can probably have a modified review out by Wednesday evening. What do you think about starting out with a simple configuration (just log_size and backups on RotatePolicy) and iterating from there? I also have one question - what distinction are you making between the standard flag on Logger and the existence of a rotation policy? Brian Wickman wrote: Yeah, all the extra parameters were just for illustration only. Not asking for any more functionality than what you already have since it already provides tremendous value. The idea for 'standard' in Logger is just to be explicit about current behavior (unrestricted logging to stdout/stderr) and use it as the default. As for applying a universal default that's not standard, there are a few ways that you could do this, from environment variables (THERMOS_FORCE_ROTATE? idk) to building an aurora client using a custom entry point that patches Process.TYPEMAP['logger'] to use a different default. Both are kind of sketch but within the realm of sketch found elsewhere in the code. The 'standard' flag makes sense to me, thanks. What do you envision reading the environment variable? The executor/runner? I suppose we could enhance the scheduler so that you can pass it environment variables to set when launching the executor so there wouldn't be a lot of plumbing. I guess in general I'm not a huge fan of using the client to enforce basic operational parameters like this (although I guess it's debatable as to whether or not these settings qualify :)). For example, it makes it much more challenging to move to a model where jobs are created/started through native API calls to the scheduler instead of using the client. - George --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/#review71472 --- On Feb. 6, 2015, 5:51 p.m., George Sirois wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/ --- (Updated Feb. 6, 2015, 5:51 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-95 https://issues.apache.org/jira/browse/AURORA-95 Repository: aurora Description --- Implements log rotation in the Thermos runner. Diffs - docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/thermos_task_runner.py 9ff8c5379aad7ac05e44b1f5a2b74f759f26 src/main/python/apache/thermos/bin/thermos_runner.py bd8cf7f4cda54b6be72dad64f9446eedeb132211 src/main/python/apache/thermos/core/process.py 5ce138dab161d880c0bd58b87a6f5a54d4ca2f99 src/main/python/apache/thermos/core/runner.py f949f279a071c6464b026749f51afc776102f2aa src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 Diff: https://reviews.apache.org/r/30695/diff/ Testing --- ./pants test src/test/python/apache/thermos/core:all
Re: Review Request 30741: Add an interface and implementations of PathDetector.
On Feb. 6, 2015, 9:44 p.m., Joshua Cohen wrote: src/main/python/apache/thermos/monitoring/detector.py, line 45 https://reviews.apache.org/r/30741/diff/2/?file=852426#file852426line45 Just for my own education, what's the difference between returning `self._paths` and `self._paths[:]`? self._paths[:] returns a copy of the list instead of the list itself. this prevents people from mutating the list and polluting subsequent calls. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/#review71505 --- On Feb. 6, 2015, 7:30 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/ --- (Updated Feb. 6, 2015, 7:30 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1024 https://issues.apache.org/jira/browse/AURORA-1024 Repository: aurora Description --- The goal here is that any place that uses a fixed checkpoint root that potentially needs to interact with multiple checkpoint roots (e.g. thermos observer, thermos cli, gc executor) will instead use a PathDetector implementation. The default implementation is just the FixedPathDetector to which you can pass --checkpoint_root. However in subsequent reviews, we will 1) plumb PathDetector into the abovementioned components and then 2) wire up a ChainedPathDetector that has both a FixedPathDetector and MesosPathDetector. Diffs - src/main/python/apache/aurora/executor/BUILD 79037bc5af67e5287afa2270b70992cac42de5a7 src/main/python/apache/aurora/executor/common/BUILD e64362e3b7d603f51b0545db414a3b3df414594f src/main/python/apache/aurora/executor/common/path_detector.py PRE-CREATION src/main/python/apache/aurora/executor/executor_detector.py 7b8fc4e1cf0b03f7d425a2f6bd0944b583c96737 src/main/python/apache/aurora/executor/gc_executor.py 952d77d8ff525ef069a921a987d056de81fc7476 src/main/python/apache/thermos/monitoring/BUILD 33259c854f7b28f772310620f9101fb304f30715 src/main/python/apache/thermos/monitoring/detector.py 117aef5b0f11dd33781b314be7c80cb4034cc9f8 src/test/python/apache/aurora/executor/BUILD aa3bc3b934ea40ee34241521e3f581b0d8ed0ab6 src/test/python/apache/aurora/executor/common/BUILD 9e3a657523fc9b848785fcbb0db2e737d371f08f src/test/python/apache/aurora/executor/common/test_path_detector.py PRE-CREATION src/test/python/apache/aurora/executor/test_executor_detector.py 85d541838c3f74afc022f4ea672e06b90e7e8142 src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/30741/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor/common:: src/test/python/apache/thermos/monitoring:: One failure due to incompatible merge w/ r/30704 -- will send follow up patch. Thanks, Brian Wickman
Re: Review Request 30225: Modifying update controller to support heartbeats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30225/ --- (Updated Feb. 7, 2015, 2:43 a.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, and Bill Farner. Bugs: AURORA-1010 https://issues.apache.org/jira/browse/AURORA-1010 Repository: aurora Description (updated) --- Added pulsing support into the JobUpdateController. The qualified coordinated updates get blocked until a pulse arrives. An update then becomes active and proceeds until `blockIfNoPulsesAfterMs` expires or the update reaches a terminal state (whichever comes first). UPDATE: - Added explicit states to capture blocked updates - Refactored pulseUpdate() to not rely on DB state - Dropped JobUpdatePulseState.PAUSED as it does not seem necessary. Diffs (updated) - api/src/main/thrift/org/apache/aurora/gen/api.thrift 4f603f9e7ed004e53937a45ea8edf7241e15f5cf src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java 7f2ec71d744d5fafac84291cc79feba3398d0e1e src/main/java/org/apache/aurora/scheduler/storage/JobUpdateStore.java b7d8d524e15f101416889c00efc3ecf2c8d9c1a1 src/main/java/org/apache/aurora/scheduler/storage/db/DBJobUpdateStore.java d479d20259f284528b2291e2e486b36d8e47fe5e src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java 60f535988a20638fb16515d5027bfa65f1afb73c src/main/java/org/apache/aurora/scheduler/updater/JobUpdateController.java d3b30d48b76d8d7c64cda006a34f7ed3296526f2 src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java ec9d1e07abca1bf30262e1c0f741a34741e96f5d src/main/java/org/apache/aurora/scheduler/updater/JobUpdateStateMachine.java 76460f95e058181b711fb6869f5a34c1d5bdfe31 src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml f9c9ceddc559b43b4a5c45c745d54ff47484edde src/test/java/org/apache/aurora/scheduler/storage/db/DBJobUpdateStoreTest.java ca7c0c2675477cc727ca006697665f997972dfde src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java 92cfa2b30c1c4daa3ae225fc8609fbeaecdaff7a src/test/java/org/apache/aurora/scheduler/updater/JobUpdateStateMachineTest.java 89765ac3d34a827d3748fb96a275d78e9d1b8b72 src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 41d422939209d0808235128e4242c11e8ef25969 Diff: https://reviews.apache.org/r/30225/diff/ Testing --- ./gradlew -Pq build Thanks, Maxim Khutornenko
Re: Review Request 30741: Add an interface and implementations of PathDetector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/#review71518 --- Ship it! Master (da296a3) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 6, 2015, 10:40 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/ --- (Updated Feb. 6, 2015, 10:40 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1024 https://issues.apache.org/jira/browse/AURORA-1024 Repository: aurora Description --- The goal here is that any place that uses a fixed checkpoint root that potentially needs to interact with multiple checkpoint roots (e.g. thermos observer, thermos cli, gc executor) will instead use a PathDetector implementation. The default implementation is just the FixedPathDetector to which you can pass --checkpoint_root. However in subsequent reviews, we will 1) plumb PathDetector into the abovementioned components and then 2) wire up a ChainedPathDetector that has both a FixedPathDetector and MesosPathDetector. Diffs - src/main/python/apache/aurora/executor/BUILD 79037bc5af67e5287afa2270b70992cac42de5a7 src/main/python/apache/aurora/executor/common/BUILD e64362e3b7d603f51b0545db414a3b3df414594f src/main/python/apache/aurora/executor/common/path_detector.py PRE-CREATION src/main/python/apache/aurora/executor/executor_detector.py 7b8fc4e1cf0b03f7d425a2f6bd0944b583c96737 src/main/python/apache/aurora/executor/gc_executor.py 952d77d8ff525ef069a921a987d056de81fc7476 src/main/python/apache/thermos/monitoring/BUILD 33259c854f7b28f772310620f9101fb304f30715 src/main/python/apache/thermos/monitoring/detector.py 117aef5b0f11dd33781b314be7c80cb4034cc9f8 src/test/python/apache/aurora/executor/BUILD aa3bc3b934ea40ee34241521e3f581b0d8ed0ab6 src/test/python/apache/aurora/executor/common/BUILD 9e3a657523fc9b848785fcbb0db2e737d371f08f src/test/python/apache/aurora/executor/common/test_path_detector.py PRE-CREATION src/test/python/apache/aurora/executor/test_executor_detector.py 85d541838c3f74afc022f4ea672e06b90e7e8142 src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/30741/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor/common:: src/test/python/apache/thermos/monitoring:: One failure due to incompatible merge w/ r/30704 -- will send follow up patch. Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71544 --- Ship it! Master (11a65d2) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 7, 2015, 2:27 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 7, 2015, 2:27 a.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - pants.ini 99648e46ca755c3c8e22d90c7b682107d8dee333 src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30741: Add an interface and implementations of PathDetector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/ --- (Updated Feb. 6, 2015, 10:40 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- address jcohen's feedback. Bugs: AURORA-1024 https://issues.apache.org/jira/browse/AURORA-1024 Repository: aurora Description --- The goal here is that any place that uses a fixed checkpoint root that potentially needs to interact with multiple checkpoint roots (e.g. thermos observer, thermos cli, gc executor) will instead use a PathDetector implementation. The default implementation is just the FixedPathDetector to which you can pass --checkpoint_root. However in subsequent reviews, we will 1) plumb PathDetector into the abovementioned components and then 2) wire up a ChainedPathDetector that has both a FixedPathDetector and MesosPathDetector. Diffs (updated) - src/main/python/apache/aurora/executor/BUILD 79037bc5af67e5287afa2270b70992cac42de5a7 src/main/python/apache/aurora/executor/common/BUILD e64362e3b7d603f51b0545db414a3b3df414594f src/main/python/apache/aurora/executor/common/path_detector.py PRE-CREATION src/main/python/apache/aurora/executor/executor_detector.py 7b8fc4e1cf0b03f7d425a2f6bd0944b583c96737 src/main/python/apache/aurora/executor/gc_executor.py 952d77d8ff525ef069a921a987d056de81fc7476 src/main/python/apache/thermos/monitoring/BUILD 33259c854f7b28f772310620f9101fb304f30715 src/main/python/apache/thermos/monitoring/detector.py 117aef5b0f11dd33781b314be7c80cb4034cc9f8 src/test/python/apache/aurora/executor/BUILD aa3bc3b934ea40ee34241521e3f581b0d8ed0ab6 src/test/python/apache/aurora/executor/common/BUILD 9e3a657523fc9b848785fcbb0db2e737d371f08f src/test/python/apache/aurora/executor/common/test_path_detector.py PRE-CREATION src/test/python/apache/aurora/executor/test_executor_detector.py 85d541838c3f74afc022f4ea672e06b90e7e8142 src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/30741/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor/common:: src/test/python/apache/thermos/monitoring:: One failure due to incompatible merge w/ r/30704 -- will send follow up patch. Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 7, 2015, 12:46 a.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Master merge. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 27182: Add a test for the thermos resource module
On Feb. 6, 2015, 1:55 p.m., Brian Wickman wrote: Out of curiosity, why this test now? I sort of see this code on its deathbed as soon Mesos will have disk enforcement built in. This test is from October (and just revived). Until we remove this code, it should have test coverage. - Joe --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/#review71510 --- On Feb. 6, 2015, 2 p.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/ --- (Updated Feb. 6, 2015, 2 p.m.) Review request for Aurora and Brian Wickman. Repository: aurora Description --- Add a test for the thermos resource module Diffs - src/main/python/apache/thermos/monitoring/monitor.py 8f87f5ffc39c87e87ff78b941ea30df7138bd1ef src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_resource.py PRE-CREATION Diff: https://reviews.apache.org/r/27182/diff/ Testing --- [tw-mbp-jsmith incubator-aurora (yasumoto/thermos_test)]$ ./pants test.pytest ./src/test/python/apache/thermos/monitoring:test_resource 21:15:56 00:00 [main] (To run a reporting server: ./pants server) 21:15:56 00:00 [bootstrap] 21:15:57 00:01 [setup] 21:15:57 00:01 [parse] Executing tasks in goals: bootstrap - imports - gen - check-exclusives - resolve - compile - resources - test 21:15:57 00:01 [bootstrap] 21:15:57 00:01 [bootstrap-jvm-tools] 21:15:57 00:01 [imports] 21:15:57 00:01 [ivy-imports] 21:15:57 00:01 [gen] 21:15:57 00:01 [thrift] 21:15:57 00:01 [scrooge] 21:15:57 00:01 [protoc] 21:15:57 00:01 [antlr] 21:15:57 00:01 [ragel] 21:15:57 00:01 [jaxb] 21:15:57 00:01 [wire] 21:15:57 00:01 [aapt] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [resolve] 21:15:57 00:01 [ivy] 21:15:57 00:01 [compile] 21:15:57 00:01 [jvm] 21:15:57 00:01 [jvm-compilers] 21:15:57 00:01 [resources] 21:15:57 00:01 [prepare] 21:15:57 00:01 [test] 21:15:57 00:01 [run_prep_command] 21:15:57 00:01 [pytest] 21:15:57 00:01 [run] == test session starts === platform darwin -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4 plugins: cov, timeout collected 5 items src/test/python/apache/thermos/monitoring/test_resource.py . 5 passed in 0.18 seconds 21:16:04 00:08 [junit] 21:16:04 00:08 [specs] SUCCESS Thanks, Joe Smith
Re: Review Request 30695: Implements log rotation in the Thermos runner.
On Feb. 6, 2015, 6:52 p.m., Brian Wickman wrote: This is super rad. Thanks for taking this on. Before I do a deeper dive, what do you think about making the logrotate policy be specified by the user instead of the framework owner, with a sensible default? For example, if this is configurable on the process object, you can have different policies per process, e.g. ```py class RotatePolicy(Struct): log_size = Default(Integer, 32*MB) backups = Default(Integer, 10) copytruncate = Default(Boolean, False) compress = Default(Boolean, False) hangup_command = String ... # union class Logger(Struct): standard = Boolean # standard i/o devnull = Boolean # /dev/null redirection logrotate = RotatePolicy # use a logrotation policy DefaultLogger = Logger(standard=True) class Process(Struct): cmdline = Required(String) name= Required(String) ... logger = Default(Logger, DefaultLogger) ``` This also means reduced end-to-end plumbing through all the binaries, class constructors, etc. And if you ever need to add new features (e.g. a compress option), they're fairly well encapsulated within the Logger union. George Sirois wrote: Awesome, thanks for the feedback. I'd be willing to take this on; it would definitely make the plumbing a lot cleaner and provide more flexibility, although the downside is that it's now harder to apply a universal default (besides whatever we arrive at as the Aurora default). I'll be able to pick this up next week and can probably have a modified review out by Wednesday evening. What do you think about starting out with a simple configuration (just log_size and backups on RotatePolicy) and iterating from there? I also have one question - what distinction are you making between the standard flag on Logger and the existence of a rotation policy? Yeah, all the extra parameters were just for illustration only. Not asking for any more functionality than what you already have since it already provides tremendous value. The idea for 'standard' in Logger is just to be explicit about current behavior (unrestricted logging to stdout/stderr) and use it as the default. As for applying a universal default that's not standard, there are a few ways that you could do this, from environment variables (THERMOS_FORCE_ROTATE? idk) to building an aurora client using a custom entry point that patches Process.TYPEMAP['logger'] to use a different default. Both are kind of sketch but within the realm of sketch found elsewhere in the code. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/#review71472 --- On Feb. 6, 2015, 5:51 p.m., George Sirois wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/ --- (Updated Feb. 6, 2015, 5:51 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-95 https://issues.apache.org/jira/browse/AURORA-95 Repository: aurora Description --- Implements log rotation in the Thermos runner. Diffs - docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/thermos_task_runner.py 9ff8c5379aad7ac05e44b1f5a2b74f759f26 src/main/python/apache/thermos/bin/thermos_runner.py bd8cf7f4cda54b6be72dad64f9446eedeb132211 src/main/python/apache/thermos/core/process.py 5ce138dab161d880c0bd58b87a6f5a54d4ca2f99 src/main/python/apache/thermos/core/runner.py f949f279a071c6464b026749f51afc776102f2aa src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 Diff: https://reviews.apache.org/r/30695/diff/ Testing --- ./pants test src/test/python/apache/thermos/core:all Thanks, George Sirois
Re: Review Request 30768: Reject None values for TaskPath
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/#review71577 --- I would love to see something in the top-level integration test that would have caught this. Lacking anything like selenium, we could take on some coupling to low-level details and construct/curl the observer URL: # Find the task ID. task_id=$(aurora_admin query --states=RUNNING -l '%taskId%' devcluster www-data hello) # Get the HTTP response code from the observer. check_url_live http://localhost:1338/task/$task_id Do you think that's reasonable? - Bill Farner On Feb. 8, 2015, 2:06 a.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/ --- (Updated Feb. 8, 2015, 2:06 a.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-1115 https://issues.apache.org/jira/browse/AURORA-1115 Repository: aurora Description --- This patch modifies TaskPath to reject None values. Diffs - src/main/python/apache/aurora/executor/thermos_task_runner.py 7b346e253677ee9b42c57782f7f67ff63b6a0083 src/main/python/apache/thermos/common/path.py 9e617051f16f4270b3958f48e0cc43706d245eec src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py PRE-CREATION src/test/python/apache/thermos/common/test_pathspec.py 3437b196d33d7c2ff6ba292ff99b6881954e7ecb Diff: https://reviews.apache.org/r/30768/diff/ Testing --- ./pants test src/test/python/apache/thermos:: ./pants test src/test/python/apache/aurora/executor:: ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Zameer Manji
Re: Review Request 30768: Reject None values for TaskPath
On Feb. 8, 2015, 5:28 p.m., Bill Farner wrote: I would love to see something in the top-level integration test that would have caught this. Lacking anything like selenium, we could take on some coupling to low-level details and construct/curl the observer URL: # Find the task ID. task_id=$(aurora_admin query --states=RUNNING -l '%taskId%' devcluster www-data hello) # Get the HTTP response code from the observer. check_url_live http://localhost:1338/task/$task_id Do you think that's reasonable? (to be explicit, i'm referring to `src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh`) - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/#review71577 --- On Feb. 8, 2015, 2:06 a.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/ --- (Updated Feb. 8, 2015, 2:06 a.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-1115 https://issues.apache.org/jira/browse/AURORA-1115 Repository: aurora Description --- This patch modifies TaskPath to reject None values. Diffs - src/main/python/apache/aurora/executor/thermos_task_runner.py 7b346e253677ee9b42c57782f7f67ff63b6a0083 src/main/python/apache/thermos/common/path.py 9e617051f16f4270b3958f48e0cc43706d245eec src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py PRE-CREATION src/test/python/apache/thermos/common/test_pathspec.py 3437b196d33d7c2ff6ba292ff99b6881954e7ecb Diff: https://reviews.apache.org/r/30768/diff/ Testing --- ./pants test src/test/python/apache/thermos:: ./pants test src/test/python/apache/aurora/executor:: ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Zameer Manji
Re: Review Request 30768: Reject None values for TaskPath
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/ --- (Updated Feb. 8, 2015, 1:34 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Changes --- Improve end to end tests to check for observer failure. Bugs: AURORA-1115 https://issues.apache.org/jira/browse/AURORA-1115 Repository: aurora Description --- This patch modifies TaskPath to reject None values. Diffs (updated) - src/main/python/apache/aurora/executor/thermos_task_runner.py 7b346e253677ee9b42c57782f7f67ff63b6a0083 src/main/python/apache/thermos/common/path.py 9e617051f16f4270b3958f48e0cc43706d245eec src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py PRE-CREATION src/test/python/apache/thermos/common/test_pathspec.py 3437b196d33d7c2ff6ba292ff99b6881954e7ecb src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 989801cfcbd19109ac140b01cd3024d70c78c829 Diff: https://reviews.apache.org/r/30768/diff/ Testing --- ./pants test src/test/python/apache/thermos:: ./pants test src/test/python/apache/aurora/executor:: ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Zameer Manji
Re: Review Request 30768: Reject None values for TaskPath
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/#review71587 --- Ship it! Master (11a65d2) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 8, 2015, 9:34 p.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/ --- (Updated Feb. 8, 2015, 9:34 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-1115 https://issues.apache.org/jira/browse/AURORA-1115 Repository: aurora Description --- This patch modifies TaskPath to reject None values. Diffs - src/main/python/apache/aurora/executor/thermos_task_runner.py 7b346e253677ee9b42c57782f7f67ff63b6a0083 src/main/python/apache/thermos/common/path.py 9e617051f16f4270b3958f48e0cc43706d245eec src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py PRE-CREATION src/test/python/apache/thermos/common/test_pathspec.py 3437b196d33d7c2ff6ba292ff99b6881954e7ecb src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 989801cfcbd19109ac140b01cd3024d70c78c829 Diff: https://reviews.apache.org/r/30768/diff/ Testing --- ./pants test src/test/python/apache/thermos:: ./pants test src/test/python/apache/aurora/executor:: ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Zameer Manji
Re: Review Request 30768: Reject None values for TaskPath
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/#review71589 --- Ship it! LGTM, just a style consistency nit. Thanks for fixing this! src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py https://reviews.apache.org/r/30768/#comment117381 This source now has 3 styles for continued lines: - indent matching the first argument - +2 indent - +4 indent AFAIK +4 indent is the agreed-upon convention. Can you fix the whole file to match that? - Bill Farner On Feb. 8, 2015, 9:34 p.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/ --- (Updated Feb. 8, 2015, 9:34 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-1115 https://issues.apache.org/jira/browse/AURORA-1115 Repository: aurora Description --- This patch modifies TaskPath to reject None values. Diffs - src/main/python/apache/aurora/executor/thermos_task_runner.py 7b346e253677ee9b42c57782f7f67ff63b6a0083 src/main/python/apache/thermos/common/path.py 9e617051f16f4270b3958f48e0cc43706d245eec src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py PRE-CREATION src/test/python/apache/thermos/common/test_pathspec.py 3437b196d33d7c2ff6ba292ff99b6881954e7ecb src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 989801cfcbd19109ac140b01cd3024d70c78c829 Diff: https://reviews.apache.org/r/30768/diff/ Testing --- ./pants test src/test/python/apache/thermos:: ./pants test src/test/python/apache/aurora/executor:: ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Zameer Manji
Re: Review Request 30710: add mesos role feature
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30710/
---
(Updated Feb. 9, 2015, 4:01 a.m.)
Review request for Aurora and Bill Farner.
Changes
---
add link to jira
Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109
Repository: aurora
Description
---
## Problems
We are from eBay platform team. Previously, we used marathon to generate
Jenkins master instance in dedicated vms and recieve resource offer from same
dedicated vms. For the details, please refer to
http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/#.VNQUuC6_SPU
Now, we found Aurora is more stable and powerful. We are moving from Marathon
to Aurora. During the move, we found there is no mesos role in Aurora now. But
we need use mesos role way to solve the problem in section Frameworks stopped
receiving offers after a while of the given url.
Here is a snippet of the problem description:
*We noticed occurred after we used Marathon to create the initial set of CI
masters. As those CI masters started registering themselves as frameworks,
Marathon stopped receiving any offers from Mesos; essentially, no new CI
masters could be launched. Let’s start with Marathon. In the DRF model, it was
unfair to treat Marathon in the same bucket/role alongside hundreds of
connected Jenkins frameworks. After launching all these Jenkins frameworks,
Marathon had a large resource share and Mesos would aggressively offer
resources to frameworks that were using little or no resources. Marathon was
placed last in priority and got starved out.*
*We decided to define a dedicated Mesos role for Marathon and to have all of
the Mesos slaves that were reserved for Jenkins master instances support that
Mesos role. Jenkins frameworks were left with the default role “*”.*This solved
the problem – Mesos offered resources per role and hence Marathon never got
starved out. A framework with a special role will get resource offers from both
slaves supporting that special role and also from the default role “*”.*
However, since we were using placement constraints, Marathon accepted resource
offers only from slaves that supported both the role and the placement
constraints.*
## Solution
So we add role feature is the source code to solve the problem in same way:
When accept a resource offer, Aurora will send back the needed resources to
Mesos with the mesos role in resource offer.
How to configure the Mesos role:
1.Add cmd option --mesos_role=${Mesos role name} when start Aurora scheduler.
We change the test cases according code change. Each changed test case is green
Merge https://github.com/zhanglong2015/incubator-aurora
Diffs
-
src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
1a158b4e0be94762ad0480e8ce74b19bacc90c97
src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java
31aa2bbaab3d97875493ad75c4d2c7c82ac7fa58
src/main/java/org/apache/aurora/scheduler/configuration/Resources.java
b5a3140e3560f790d1db496dca3c2ee0dc96a195
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
d0994203b5650f44ca2eb32e1e2aa61875163854
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
5340d651b298ec8aa079e73d6d2f652fdf876293
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
e1c29747c9854cf75bf63f6f085cf40ca68989af
src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java
422d5a9a42310979752eb7282658316c2b772419
src/test/java/org/apache/aurora/scheduler/configuration/ResourcesTest.java
d6febb8998e05257cabe8d193cefa0b6c79f197e
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
5f08d00d39f016af9bc296e517ad49b66ab5a8de
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
411a55a8d85f60bb2703468f2d69b64b2736eee4
Diff: https://reviews.apache.org/r/30710/diff/
Testing
---
:buildSrc:compileJava UP-TO-DATE
:buildSrc:compileGroovy UP-TO-DATE
:buildSrc:processResources UP-TO-DATE
:buildSrc:classes UP-TO-DATE
:buildSrc:jar UP-TO-DATE
:buildSrc:assemble UP-TO-DATE
:buildSrc:compileTestJava UP-TO-DATE
:buildSrc:compileTestGroovy UP-TO-DATE
:buildSrc:processTestResources UP-TO-DATE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test UP-TO-DATE
:buildSrc:check UP-TO-DATE
:buildSrc:build UP-TO-DATE
:api:generateThriftJava
:api:classesThrift
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
:api:checkPython
:api:generateThriftEntitiesJava
:api:classesThriftEntities
:api:compileJava UP-TO-DATE
:api:generateThriftResources
:api:processResources UP-TO-DATE
:api:classes
:api:jar
:compileJava
Note: Writing
