Re: Review Request 32373: Add pesos executor build target
-0.0.0-py2-none-any.whl api.src.main.thrift.org.apache.aurora.gen.py_thrift_test_d991619f-0.0.0-py2-none-any.whl api.src.main.thrift.org.apache.thermos.py_thrift_b45199dc-0.0.0-py2-none-any.whl argh-0.26.1-py2-none-any.whl backports.ssl_match_hostname-3.4.0.2-py2-none-any.whl certifi-14.05.14-py2.py3-none-any.whl compactor-0.2.0-py2.py3-none-any.whl futures-2.1.6-py2.py3-none-any.whl kazoo-1.3.1-py2-none-any.whl mesos.interface-0.21.1-py2-none-any.whl pathtools-0.1.2-py2-none-any.whl pesos-0.2.0_dev0-py2.py3-none-any.whl pex-0.8.2-py2.py3-none-any.whl protobuf-2.6.1-py2.7.egg psutil-2.1.3-cp27-none-macosx_10_9_intel.whl pystachio-0.7.4-py2.py3-none-any.whl requests-2.3.0-py2.py3-none-any.whl setuptools-7.0-py2.py3-none-any.whl thrift-0.9.1-cp27-none-macosx_10_9_intel.whl tornado-4.1-cp27-none-macosx_10_9_intel.whl trollius-1.0.4-py27-none-any.whl twitter.common.app-0.3.3-py2-none-any.whl twitter.common.collections-0.3.3-py2-none-any.whl twitter.common.concurrent-0.3.3-py2-none-any.whl twitter.common.contextutil-0.3.3-py2-none-any.whl twitter.common.decorators-0.3.3-py2-none-any.whl twitter.common.dirutil-0.3.3-py2-none-any.whl twitter.common.exceptions-0.3.3-py2-none-any.whl twitter.common.lang-0.3.3-py2-none-any.whl twitter.common.log-0.3.3-py2-none-any.whl twitter.common.metrics-0.3.3-py2-none-any.whl twitter.common.options-0.3.3-py2-none-any.whl twitter.common.process-0.3.3-py2-none-any.whl twitter.common.quantity-0.3.3-py2-none-any.whl twitter.common.recordio-0.3.3-py2-none-any.whl twitter.common.string-0.3.3-py2-none-any.whl twitter.common.util-0.3.3-py2-none-any.whl twitter.common.zookeeper-0.3.3-py2-none-any.whl watchdog-0.7.0-cp27-none-macosx_10_9_intel.whl zope.interface-4.1.2-cp27-none-macosx_10_9_intel.whl Thanks, Brian Wickman
Review Request 32515: Kill inotify disk collector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32515/ --- Review request for Aurora, Kevin Sweeney and Zameer Manji. Repository: aurora Description --- This is currently unwired and its only test is marked flaky. Diffs - 3rdparty/python/requirements.txt ee30136c9224a85b65a93185ec12a86d7051e354 src/main/python/apache/thermos/monitoring/BUILD a7864d4e490784fef04582d4986c2fd121947c9d src/main/python/apache/thermos/monitoring/disk.py ccf6d6aafe893836e72e179cc1a8010a1ac6ac7f src/test/python/apache/thermos/monitoring/test_disk.py e1373234feba08ab881179af0cbf6b7ed326cc45 Diff: https://reviews.apache.org/r/32515/diff/ Testing --- Thanks, Brian Wickman
Re: Review Request 32515: Kill inotify disk collector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32515/ --- (Updated March 26, 2015, 1:33 a.m.) Review request for Aurora, Kevin Sweeney and Zameer Manji. Changes --- Fix checkstyle violations. Repository: aurora Description --- This is currently unwired and its only test is marked flaky. Diffs (updated) - 3rdparty/python/requirements.txt ee30136c9224a85b65a93185ec12a86d7051e354 src/main/python/apache/thermos/monitoring/BUILD a7864d4e490784fef04582d4986c2fd121947c9d src/main/python/apache/thermos/monitoring/disk.py ccf6d6aafe893836e72e179cc1a8010a1ac6ac7f src/test/python/apache/thermos/monitoring/test_disk.py e1373234feba08ab881179af0cbf6b7ed326cc45 Diff: https://reviews.apache.org/r/32515/diff/ Testing --- Thanks, Brian Wickman
Re: Review Request 32515: Kill inotify disk collector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32515/ --- (Updated March 26, 2015, 1:55 a.m.) Review request for Aurora, Kevin Sweeney and Zameer Manji. Changes --- Add missing twitter.common.quantity for test_disk. Repository: aurora Description --- This is currently unwired and its only test is marked flaky. Diffs (updated) - 3rdparty/python/requirements.txt ee30136c9224a85b65a93185ec12a86d7051e354 src/main/python/apache/thermos/monitoring/BUILD a7864d4e490784fef04582d4986c2fd121947c9d src/main/python/apache/thermos/monitoring/disk.py ccf6d6aafe893836e72e179cc1a8010a1ac6ac7f src/test/python/apache/thermos/monitoring/BUILD 09deebb786ab3c9f4d7b88211c3f583cdc5ac9a9 src/test/python/apache/thermos/monitoring/test_disk.py e1373234feba08ab881179af0cbf6b7ed326cc45 Diff: https://reviews.apache.org/r/32515/diff/ Testing --- Thanks, Brian Wickman
Re: Review Request 32373: Add pesos executor build target
api.src.main.thrift.org.apache.thermos.py_thrift_b45199dc-0.0.0-py2-none-any.whl argh-0.26.1-py2-none-any.whl backports.ssl_match_hostname-3.4.0.2-py2-none-any.whl certifi-14.05.14-py2.py3-none-any.whl compactor-0.2.0-py2.py3-none-any.whl futures-2.1.6-py2.py3-none-any.whl kazoo-1.3.1-py2-none-any.whl mesos.interface-0.21.1-py2-none-any.whl pathtools-0.1.2-py2-none-any.whl pesos-0.2.0_dev0-py2.py3-none-any.whl pex-0.8.2-py2.py3-none-any.whl protobuf-2.6.1-py2.7.egg psutil-2.1.3-cp27-none-macosx_10_9_intel.whl pystachio-0.7.4-py2.py3-none-any.whl requests-2.3.0-py2.py3-none-any.whl setuptools-7.0-py2.py3-none-any.whl thrift-0.9.1-cp27-none-macosx_10_9_intel.whl tornado-4.1-cp27-none-macosx_10_9_intel.whl trollius-1.0.4-py27-none-any.whl twitter.common.app-0.3.3-py2-none-any.whl twitter.common.collections-0.3.3-py2-none-any.whl twitter.common.concurrent-0.3.3-py2-none-any.whl twitter.common.contextutil-0.3.3-py2-none-any.whl twitter.common.decorators-0.3.3-py2-none-any.whl twitter.common.dirutil-0.3.3-py2-none-any.whl twitter.common.exceptions-0.3.3-py2-none-any.whl twitter.common.lang-0.3.3-py2-none-any.whl twitter.common.log-0.3.3-py2-none-any.whl twitter.common.metrics-0.3.3-py2-none-any.whl twitter.common.options-0.3.3-py2-none-any.whl twitter.common.process-0.3.3-py2-none-any.whl twitter.common.quantity-0.3.3-py2-none-any.whl twitter.common.recordio-0.3.3-py2-none-any.whl twitter.common.string-0.3.3-py2-none-any.whl twitter.common.util-0.3.3-py2-none-any.whl twitter.common.zookeeper-0.3.3-py2-none-any.whl watchdog-0.7.0-cp27-none-macosx_10_9_intel.whl zope.interface-4.1.2-cp27-none-macosx_10_9_intel.whl Thanks, Brian Wickman
Review Request 32373: Add pesos executor build target
compactor-0.2.0-py2.py3-none-any.whl futures-2.1.6-py2.py3-none-any.whl kazoo-1.3.1-py2-none-any.whl mesos.interface-0.21.1-py2-none-any.whl pathtools-0.1.2-py2-none-any.whl pesos-0.2.0_dev0-py2.py3-none-any.whl pex-0.8.2-py2.py3-none-any.whl protobuf-2.6.1-py2.7.egg psutil-2.1.3-cp27-none-macosx_10_9_intel.whl pystachio-0.7.4-py2.py3-none-any.whl requests-2.3.0-py2.py3-none-any.whl setuptools-7.0-py2.py3-none-any.whl thrift-0.9.1-cp27-none-macosx_10_9_intel.whl tornado-4.1-cp27-none-macosx_10_9_intel.whl trollius-1.0.4-py27-none-any.whl twitter.common.app-0.3.3-py2-none-any.whl twitter.common.collections-0.3.3-py2-none-any.whl twitter.common.concurrent-0.3.3-py2-none-any.whl twitter.common.contextutil-0.3.3-py2-none-any.whl twitter.common.decorators-0.3.3-py2-none-any.whl twitter.common.dirutil-0.3.3-py2-none-any.whl twitter.common.exceptions-0.3.3-py2-none-any.whl twitter.common.lang-0.3.3-py2-none-any.whl twitter.common.log-0.3.3-py2-none-any.whl twitter.common.metrics-0.3.3-py2-none-any.whl twitter.common.options-0.3.3-py2-none-any.whl twitter.common.process-0.3.3-py2-none-any.whl twitter.common.quantity-0.3.3-py2-none-any.whl twitter.common.recordio-0.3.3-py2-none-any.whl twitter.common.string-0.3.3-py2-none-any.whl twitter.common.util-0.3.3-py2-none-any.whl twitter.common.zookeeper-0.3.3-py2-none-any.whl watchdog-0.7.0-cp27-none-macosx_10_9_intel.whl zope.interface-4.1.2-cp27-none-macosx_10_9_intel.whl Thanks, Brian Wickman
Re: Review Request 32221: Remove excessively low timeout in SIGTERM swallowing test.
On March 18, 2015, 11:01 p.m., Aurora ReviewBot wrote: Master (6396410) is red with this patch. ./build-support/jenkins/build.sh src.test.python.apache.aurora.client.cli.plugins . SUCCESS src.test.python.apache.aurora.client.cli.quota . SUCCESS src.test.python.apache.aurora.client.cli.sla . SUCCESS src.test.python.apache.aurora.client.cli.supdate . SUCCESS src.test.python.apache.aurora.client.cli.task . SUCCESS src.test.python.apache.aurora.client.cli.update . SUCCESS src.test.python.apache.aurora.client.cli.version . SUCCESS src.test.python.apache.aurora.client.config . SUCCESS src.test.python.apache.aurora.client.factory . SUCCESS src.test.python.apache.aurora.client.hooks.hooked_api . SUCCESS src.test.python.apache.aurora.client.hooks.non_hooked_api . SUCCESS src.test.python.apache.aurora.common.test_aurora_job_key . SUCCESS src.test.python.apache.aurora.common.test_cluster . SUCCESS src.test.python.apache.aurora.common.test_cluster_option . SUCCESS src.test.python.apache.aurora.common.test_clusters . SUCCESS src.test.python.apache.aurora.common.test_http_signaler . SUCCESS src.test.python.apache.aurora.common.test_pex_version . SUCCESS src.test.python.apache.aurora.common.test_shellify . SUCCESS src.test.python.apache.aurora.common.test_transport . SUCCESS src.test.python.apache.aurora.config.test_base . SUCCESS src.test.python.apache.aurora.config.test_constraint_parsing . SUCCESS src.test.python.apache.aurora.config.test_loader . SUCCESS src.test.python.apache.aurora.config.test_thrift . SUCCESS src.test.python.apache.aurora.executor.common.path_detector . SUCCESS src.test.python.apache.aurora.executor.common.task_info . SUCCESS src.test.python.apache.aurora.executor.executor_base . SUCCESS src.test.python.apache.aurora.executor.executor_vars . SUCCESS src.test.python.apache.aurora.executor.status_manager . SUCCESS src.test.python.apache.aurora.executor.thermos_task_runner . FAILURE src.test.python.apache.thermos.cli.commands.commands . SUCCESS src.test.python.apache.thermos.cli.common . SUCCESS src.test.python.apache.thermos.cli.main . SUCCESS src.test.python.apache.thermos.common.test_pathspec . SUCCESS src.test.python.apache.thermos.core.test_runner_integration . SUCCESS src.test.python.apache.thermos.monitoring.test_disk . SUCCESS FAILURE [31m FAILURE[0m I will refresh this build result if you post a review containing @ReviewBot retry Brian Wickman wrote: welp my kingdom for reviewbot to print out stderr logs - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32221/#review76965 --- On March 18, 2015, 10:44 p.m., Brian Wickman wrote
Re: Review Request 32221: Remove excessively low timeout in SIGTERM swallowing test.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32221/#review76992 --- @ReviewBot retry - Brian Wickman On March 19, 2015, 1:20 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32221/ --- (Updated March 19, 2015, 1:20 a.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-1054 https://issues.apache.org/jira/browse/AURORA-1054 Repository: aurora Description --- Remove excessively low timeout in SIGTERM swallowing test. Diffs - src/test/python/apache/aurora/executor/test_thermos_task_runner.py 6b24bbb2ab7ca16f97961aabeed945b61e5b5908 Diff: https://reviews.apache.org/r/32221/diff/ Testing --- Cannot reproduce locally, but 5 seconds is an impossibly small timeout, even if we aren't testing SIGTERM swallowing. If this fails, we will get tripped by 60s timeout instead. Thanks, Brian Wickman
Re: Review Request 32221: Remove excessively low timeout in SIGTERM swallowing test.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32221/ --- (Updated March 19, 2015, 1:20 a.m.) Review request for Aurora and Bill Farner. Changes --- The low timeout is actually what allows the test to even work in the first place. It should've worked in the past, but at least this way we can possibly eke out the original reason it was failing. Bugs: AURORA-1054 https://issues.apache.org/jira/browse/AURORA-1054 Repository: aurora Description --- Remove excessively low timeout in SIGTERM swallowing test. Diffs (updated) - src/test/python/apache/aurora/executor/test_thermos_task_runner.py 6b24bbb2ab7ca16f97961aabeed945b61e5b5908 Diff: https://reviews.apache.org/r/32221/diff/ Testing --- Cannot reproduce locally, but 5 seconds is an impossibly small timeout, even if we aren't testing SIGTERM swallowing. If this fails, we will get tripped by 60s timeout instead. Thanks, Brian Wickman
Re: Review Request 32221: Remove excessively low timeout in SIGTERM swallowing test.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32221/#review76998 --- @ReviewBot retry - Brian Wickman On March 19, 2015, 1:20 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32221/ --- (Updated March 19, 2015, 1:20 a.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-1054 https://issues.apache.org/jira/browse/AURORA-1054 Repository: aurora Description --- Remove excessively low timeout in SIGTERM swallowing test. Diffs - src/test/python/apache/aurora/executor/test_thermos_task_runner.py 6b24bbb2ab7ca16f97961aabeed945b61e5b5908 Diff: https://reviews.apache.org/r/32221/diff/ Testing --- Cannot reproduce locally, but 5 seconds is an impossibly small timeout, even if we aren't testing SIGTERM swallowing. If this fails, we will get tripped by 60s timeout instead. Thanks, Brian Wickman
Review Request 32221: Remove excessively low timeout in SIGTERM swallowing test.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32221/ --- Review request for Aurora and Bill Farner. Bugs: AURORA-1054 https://issues.apache.org/jira/browse/AURORA-1054 Repository: aurora Description --- Remove excessively low timeout in SIGTERM swallowing test. Diffs - src/test/python/apache/aurora/executor/test_thermos_task_runner.py 6b24bbb2ab7ca16f97961aabeed945b61e5b5908 Diff: https://reviews.apache.org/r/32221/diff/ Testing --- Cannot reproduce locally, but 5 seconds is an impossibly small timeout, even if we aren't testing SIGTERM swallowing. If this fails, we will get tripped by 60s timeout instead. Thanks, Brian Wickman
Re: Review Request 32013: Cache TaskDetectors for reuse.
On March 17, 2015, 7:49 p.m., Zameer Manji wrote: src/main/python/apache/aurora/executor/common/path_detector.py, line 34 https://reviews.apache.org/r/32013/diff/3/?file=896451#file896451line34 What's the rationale behind this change? I think the callers should be doing the checking for existance and dealing with missing directories as needed. The rationale is that some tasks will have checkpointing in the sandbox (0.9.0+) and some will have it outside the sandbox (0.8.0, or 0.9.0+ but whose thermos runners haven't yet been forked.) A typical production slave has 1000-2000 leftover framework directories, but since we haven't rolled sandbox checkpointing yet, that's 1000-2000 TaskDetectors we must construct and query if we don't filter by tasks that have this feature enabled. That filtering definitely seems like it belongs in the MesosPathDetector. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/#review76784 --- On March 17, 2015, 8:24 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 17, 2015, 8:24 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- SSIA Diffs - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c src/test/python/apache/aurora/executor/common/test_path_detector.py 2d6edae420c1c5b51fe93f97e04c24976afacd51 Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Re: Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 17, 2015, 8:24 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- rebase against master Repository: aurora Description --- SSIA Diffs (updated) - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c src/test/python/apache/aurora/executor/common/test_path_detector.py 2d6edae420c1c5b51fe93f97e04c24976afacd51 Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Re: Review Request 32131: Get CI to build python sdists.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32131/#review76684 --- src/main/python/apache/thermos/bin/BUILD https://reviews.apache.org/r/32131/#comment124315 only a minor concern that people depend on this, but it's probably ok to remove. - Brian Wickman On March 16, 2015, 10:18 p.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32131/ --- (Updated March 16, 2015, 10:18 p.m.) Review request for Aurora and Brian Wickman. Bugs: AURORA-1174 https://issues.apache.org/jira/browse/AURORA-1174 Repository: aurora Description --- Get CI to build python sdists. Diffs - build-support/jenkins/build.sh 850e08b1b82f368d452e3928eed8cbaadb418449 src/main/python/apache/thermos/BUILD 0dc035f759dd9949997f0c979b3556a350cf8df7 src/main/python/apache/thermos/bin/BUILD 669f9930a3590184dc0f8b5c15c36168e715eb03 Diff: https://reviews.apache.org/r/32131/diff/ Testing --- ./build-support/jenkins/build.sh Thanks, Zameer Manji
Re: Review Request 32012: Improve TaskDetector performance by memoizing regular expressions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32012/#review76586 --- @ReviewBot retry - Brian Wickman On March 13, 2015, 7:38 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32012/ --- (Updated March 13, 2015, 7:38 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- We found that the new observer burns 10x the cpu in production. Most of the time is spent in compiling new regular expressions, so memoize them. Follow-up review adds TaskDetector caching in observer. Diffs - src/main/python/apache/thermos/common/path.py 4359a5143551fff3cf66e39e68ba9ba02b8d6e2e src/main/python/apache/thermos/monitoring/detector.py e0922e61ef03fc578988f992aba46570ea1b5d6c src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 src/test/python/apache/thermos/monitoring/BUILD 4e99da403f322940579fd4f9412e022df0790884 src/test/python/apache/thermos/monitoring/test_detector.py e9005c402ec81bbb415d9c814aa7b4e759138304 Diff: https://reviews.apache.org/r/32012/diff/ Testing --- Added generic TaskDetector test and made sure it passes before/after. Thanks, Brian Wickman
Re: Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 16, 2015, 9:47 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- Patch os.path.exists in the executor path detector test. Repository: aurora Description --- SSIA Diffs (updated) - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/common/path.py 4359a5143551fff3cf66e39e68ba9ba02b8d6e2e src/main/python/apache/thermos/monitoring/detector.py e0922e61ef03fc578988f992aba46570ea1b5d6c src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c src/test/python/apache/aurora/executor/common/test_path_detector.py 2d6edae420c1c5b51fe93f97e04c24976afacd51 src/test/python/apache/thermos/monitoring/BUILD 4e99da403f322940579fd4f9412e022df0790884 src/test/python/apache/thermos/monitoring/test_detector.py e9005c402ec81bbb415d9c814aa7b4e759138304 Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Re: Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 16, 2015, 10:37 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- Incorrectly parented the last review off master instead of 32012 Repository: aurora Description --- SSIA Diffs (updated) - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c src/test/python/apache/aurora/executor/common/test_path_detector.py 2d6edae420c1c5b51fe93f97e04c24976afacd51 src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Re: Review Request 32012: Improve TaskDetector performance by memoizing regular expressions
On March 13, 2015, 1:10 a.m., Brian Wickman wrote: @ReviewBot retry Bill Farner wrote: If you believe you have discovered a flaky test (which i assume you're asking the bot to try again), can you please file a ticket? Not a flaky test -- a legitimate breakage that I did not anticipate. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32012/#review76322 --- On March 13, 2015, 12:47 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32012/ --- (Updated March 13, 2015, 12:47 a.m.) Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- We found that the new observer burns 10x the cpu in production. Most of the time is spent in compiling new regular expressions, so memoize them. Follow-up review adds TaskDetector caching in observer. Diffs - src/main/python/apache/thermos/common/path.py 4359a5143551fff3cf66e39e68ba9ba02b8d6e2e src/main/python/apache/thermos/monitoring/detector.py e0922e61ef03fc578988f992aba46570ea1b5d6c src/test/python/apache/thermos/monitoring/BUILD 4e99da403f322940579fd4f9412e022df0790884 src/test/python/apache/thermos/monitoring/test_detector.py e9005c402ec81bbb415d9c814aa7b4e759138304 Diff: https://reviews.apache.org/r/32012/diff/ Testing --- Added generic TaskDetector test and made sure it passes before/after. Thanks, Brian Wickman
Review Request 32012: Improve TaskDetector performance by memoizing regular expressions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32012/ --- Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- We found that the new observer burns 10x the cpu in production. Most of the time is spent in compiling new regular expressions, so memoize them. Follow-up review adds TaskDetector caching in observer. Diffs - src/main/python/apache/thermos/common/path.py 4359a5143551fff3cf66e39e68ba9ba02b8d6e2e src/main/python/apache/thermos/monitoring/detector.py e0922e61ef03fc578988f992aba46570ea1b5d6c src/test/python/apache/thermos/monitoring/BUILD 4e99da403f322940579fd4f9412e022df0790884 src/test/python/apache/thermos/monitoring/test_detector.py e9005c402ec81bbb415d9c814aa7b4e759138304 Diff: https://reviews.apache.org/r/32012/diff/ Testing --- Added generic TaskDetector test and made sure it passes before/after. Thanks, Brian Wickman
Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- SSIA Diffs - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Re: Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/#review76319 --- whoops, forgot that I added os.path.exists filtering -- will update tests. - Brian Wickman On March 13, 2015, 12:48 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 13, 2015, 12:48 a.m.) Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- SSIA Diffs - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Re: Review Request 32012: Improve TaskDetector performance by memoizing regular expressions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32012/#review76322 --- @ReviewBot retry - Brian Wickman On March 13, 2015, 12:47 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32012/ --- (Updated March 13, 2015, 12:47 a.m.) Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- We found that the new observer burns 10x the cpu in production. Most of the time is spent in compiling new regular expressions, so memoize them. Follow-up review adds TaskDetector caching in observer. Diffs - src/main/python/apache/thermos/common/path.py 4359a5143551fff3cf66e39e68ba9ba02b8d6e2e src/main/python/apache/thermos/monitoring/detector.py e0922e61ef03fc578988f992aba46570ea1b5d6c src/test/python/apache/thermos/monitoring/BUILD 4e99da403f322940579fd4f9412e022df0790884 src/test/python/apache/thermos/monitoring/test_detector.py e9005c402ec81bbb415d9c814aa7b4e759138304 Diff: https://reviews.apache.org/r/32012/diff/ Testing --- Added generic TaskDetector test and made sure it passes before/after. Thanks, Brian Wickman
Re: Review Request 30695: Implements log rotation in the Thermos runner.
On Feb. 6, 2015, 6:52 p.m., Brian Wickman wrote: This is super rad. Thanks for taking this on. Before I do a deeper dive, what do you think about making the logrotate policy be specified by the user instead of the framework owner, with a sensible default? For example, if this is configurable on the process object, you can have different policies per process, e.g. ```py class RotatePolicy(Struct): log_size = Default(Integer, 32*MB) backups = Default(Integer, 10) copytruncate = Default(Boolean, False) compress = Default(Boolean, False) hangup_command = String ... # union class Logger(Struct): standard = Boolean # standard i/o devnull = Boolean # /dev/null redirection logrotate = RotatePolicy # use a logrotation policy DefaultLogger = Logger(standard=True) class Process(Struct): cmdline = Required(String) name= Required(String) ... logger = Default(Logger, DefaultLogger) ``` This also means reduced end-to-end plumbing through all the binaries, class constructors, etc. And if you ever need to add new features (e.g. a compress option), they're fairly well encapsulated within the Logger union. George Sirois wrote: Awesome, thanks for the feedback. I'd be willing to take this on; it would definitely make the plumbing a lot cleaner and provide more flexibility, although the downside is that it's now harder to apply a universal default (besides whatever we arrive at as the Aurora default). I'll be able to pick this up next week and can probably have a modified review out by Wednesday evening. What do you think about starting out with a simple configuration (just log_size and backups on RotatePolicy) and iterating from there? I also have one question - what distinction are you making between the standard flag on Logger and the existence of a rotation policy? Brian Wickman wrote: Yeah, all the extra parameters were just for illustration only. Not asking for any more functionality than what you already have since it already provides tremendous value. The idea for 'standard' in Logger is just to be explicit about current behavior (unrestricted logging to stdout/stderr) and use it as the default. As for applying a universal default that's not standard, there are a few ways that you could do this, from environment variables (THERMOS_FORCE_ROTATE? idk) to building an aurora client using a custom entry point that patches Process.TYPEMAP['logger'] to use a different default. Both are kind of sketch but within the realm of sketch found elsewhere in the code. George Sirois wrote: The 'standard' flag makes sense to me, thanks. What do you envision reading the environment variable? The executor/runner? I suppose we could enhance the scheduler so that you can pass it environment variables to set when launching the executor so there wouldn't be a lot of plumbing. I guess in general I'm not a huge fan of using the client to enforce basic operational parameters like this (although I guess it's debatable as to whether or not these settings qualify :)). For example, it makes it much more challenging to move to a model where jobs are created/started through native API calls to the scheduler instead of using the client. Sorry, I totally missed this follow-up comment. If you want to enforce defaults with the client out of the picture, then probably the best way to do this is to still implement the plumbing as described above but omit Default(Logger, DefaultLogger), letting it be Empty by default. Add command line parameters to thermos_runner that allow you to toggle which logger is the default (e.g. --process_logger='rotate' --rotate_log_size=...) With this in place, you can create a new TaskRunnerProvider (TellApartThermosTaskRunnerProvider? :-) or add flags to the default one that get plumbed through to the aurora_executor command line. (e.g. --default_process_logger=rotate) This at least allows you to set organization-wide policy and will be future-proof if/when the client goes the way of the dodo in favor of a REST API. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/#review71472 --- On Feb. 6, 2015, 5:51 p.m., George Sirois wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/ --- (Updated Feb. 6, 2015, 5:51 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-95 https://issues.apache.org/jira/browse/AURORA-95 Repository: aurora Description
Re: Review Request 31774: Add aurora-specific entry point for thermos observer and fix /vars
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31774/ --- (Updated March 9, 2015, 9:24 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- remove the EndpointTracePlugin which is unused Bugs: AURORA-1133 https://issues.apache.org/jira/browse/AURORA-1133 Repository: aurora Description --- Just like the aurora-specific thermos entry point, adds an aurora-specific thermos_observer entry point. Just allows it to search for thermos runners in /var/lib/thermos and also relative to mesos sandboxes. Diffs (updated) - src/main/python/apache/aurora/tools/BUILD 36117b4199bc52873ca3abdb1ef6447437a81bb0 src/main/python/apache/aurora/tools/thermos_observer.py PRE-CREATION src/main/python/apache/thermos/observer/BUILD 28995b99b4991e17eb977583388c89e753055e9b src/main/python/apache/thermos/observer/bin/BUILD a42dbf321b8dcf5d64b22c2a480c3f4e3bad2a42 src/main/python/apache/thermos/observer/bin/thermos_observer.py 213a48eb4e2441b88fd6b608d1f3ba7dd0f2b859 src/main/python/apache/thermos/observer/http/BUILD cc8eb7793b980a6a4b76deece759e12e9bc7fcb0 src/main/python/apache/thermos/observer/http/configure.py PRE-CREATION src/main/python/apache/thermos/observer/http/diagnostics.py PRE-CREATION src/main/python/apache/thermos/observer/http/vars_endpoint.py PRE-CREATION src/main/python/apache/thermos/observer/task_observer.py 6e7517b9f1b70cef8b0400cd7769fbbe7495dc42 Diff: https://reviews.apache.org/r/31774/diff/ Testing --- mba=aurora=; ./pants test.pytest --no-fast src/test3/python:: Thanks, Brian Wickman
Re: Review Request 31559: Split thermos cli into pieces. Add custom aurora thermos entry point.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31559/ --- (Updated March 4, 2015, 6:52 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- Update diff post master merge. Bugs: AURORA-1027 https://issues.apache.org/jira/browse/AURORA-1027 Repository: aurora Description --- This splits src/main/python/apache/thermos/bin/thermos.py into src/main/python/apache/thermos/cli/**. This also adds a path detector registry (AURORA-1027) for the client so that we can easily add a customized aurora entry point for thermos that is capable of finding checkpoint roots in executor sandboxes. Added said entry point in src/main/apache/aurora/tools/thermos.py -- open for other suggestions where this should live (possibly under aurora/executor/bin?) This is a drop-in replacement for the existing thermos binary. The cli/commands/*.py are mostly no-op splits with only a few lines changed in gc, status and tail to get_path_detector. Added a basic testing framework and an example test for simplerun but did not go with full coverage, instead opting to ship AURORA-1027 for now. Diffs (updated) - src/main/python/apache/aurora/tools/BUILD PRE-CREATION src/main/python/apache/aurora/tools/thermos.py PRE-CREATION src/main/python/apache/thermos/bin/BUILD 1a002da4f8294b3ecd43ab845eb15bc8ebd12b26 src/main/python/apache/thermos/bin/thermos.py b8a638822f44cfc596b16483a19efdeb4507758f src/main/python/apache/thermos/cli/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/thermos.py PRE-CREATION src/main/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/main/python/apache/thermos/cli/commands/gc.py PRE-CREATION src/main/python/apache/thermos/cli/commands/help.py PRE-CREATION src/main/python/apache/thermos/cli/commands/inspect.py PRE-CREATION src/main/python/apache/thermos/cli/commands/kill.py PRE-CREATION src/main/python/apache/thermos/cli/commands/read.py PRE-CREATION src/main/python/apache/thermos/cli/commands/run.py PRE-CREATION src/main/python/apache/thermos/cli/commands/simplerun.py PRE-CREATION src/main/python/apache/thermos/cli/commands/status.py PRE-CREATION src/main/python/apache/thermos/cli/commands/tail.py PRE-CREATION src/main/python/apache/thermos/cli/common.py PRE-CREATION src/main/python/apache/thermos/cli/main.py PRE-CREATION src/test/python/apache/thermos/BUILD df50964851c6be89c646015edf91d07992743497 src/test/python/apache/thermos/bin/BUILD 596aeb200dc55b077ed73fc342b9187045732890 src/test/python/apache/thermos/bin/test_thermos.py ae15571948606f40bb360136311d22f52685517c src/test/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/test/python/apache/thermos/cli/commands/test_import.py PRE-CREATION src/test/python/apache/thermos/cli/commands/test_simplerun.py PRE-CREATION src/test/python/apache/thermos/cli/test_common.py PRE-CREATION Diff: https://reviews.apache.org/r/31559/diff/ Testing --- ./pants test src/test/python/apache/thermos/cli:: Thanks, Brian Wickman
Re: Review Request 31559: Split thermos cli into pieces. Add custom aurora thermos entry point.
On March 1, 2015, 9:03 p.m., Joe Smith wrote: src/main/python/apache/aurora/tools/thermos.py, line 1 https://reviews.apache.org/r/31559/diff/5/?file=881006#file881006line1 Add apache copyright stanzas to these files? done On March 1, 2015, 9:03 p.m., Joe Smith wrote: src/test/python/apache/thermos/cli/test_common.py, line 3 https://reviews.apache.org/r/31559/diff/5/?file=881031#file881031line3 Adding tests for `get_task_from_options` and `task_from_re` would be valuable added test for get_task_from_options since it's the most complex. task_from_re is simpler and the cost to cover is pretty high, so i'd like to elect to punt until we're sure it's sticking around. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31559/#review74711 --- On March 3, 2015, 10:14 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31559/ --- (Updated March 3, 2015, 10:14 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Bugs: AURORA-1027 https://issues.apache.org/jira/browse/AURORA-1027 Repository: aurora Description --- This splits src/main/python/apache/thermos/bin/thermos.py into src/main/python/apache/thermos/cli/**. This also adds a path detector registry (AURORA-1027) for the client so that we can easily add a customized aurora entry point for thermos that is capable of finding checkpoint roots in executor sandboxes. Added said entry point in src/main/apache/aurora/tools/thermos.py -- open for other suggestions where this should live (possibly under aurora/executor/bin?) This is a drop-in replacement for the existing thermos binary. The cli/commands/*.py are mostly no-op splits with only a few lines changed in gc, status and tail to get_path_detector. Added a basic testing framework and an example test for simplerun but did not go with full coverage, instead opting to ship AURORA-1027 for now. Diffs - src/main/python/apache/aurora/tools/BUILD PRE-CREATION src/main/python/apache/aurora/tools/thermos.py PRE-CREATION src/main/python/apache/thermos/bin/BUILD 1a002da4f8294b3ecd43ab845eb15bc8ebd12b26 src/main/python/apache/thermos/bin/thermos.py 0853a9892399824385bee9e72db4c108f46fceda src/main/python/apache/thermos/cli/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/thermos.py PRE-CREATION src/main/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/main/python/apache/thermos/cli/commands/gc.py PRE-CREATION src/main/python/apache/thermos/cli/commands/help.py PRE-CREATION src/main/python/apache/thermos/cli/commands/inspect.py PRE-CREATION src/main/python/apache/thermos/cli/commands/kill.py PRE-CREATION src/main/python/apache/thermos/cli/commands/read.py PRE-CREATION src/main/python/apache/thermos/cli/commands/run.py PRE-CREATION src/main/python/apache/thermos/cli/commands/simplerun.py PRE-CREATION src/main/python/apache/thermos/cli/commands/status.py PRE-CREATION src/main/python/apache/thermos/cli/commands/tail.py PRE-CREATION src/main/python/apache/thermos/cli/common.py PRE-CREATION src/main/python/apache/thermos/cli/main.py PRE-CREATION src/test/python/apache/thermos/BUILD df50964851c6be89c646015edf91d07992743497 src/test/python/apache/thermos/bin/BUILD 596aeb200dc55b077ed73fc342b9187045732890 src/test/python/apache/thermos/bin/test_thermos.py 2d9d33397ae01b31ab011d958f5457311ef7ef20 src/test/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/test/python/apache/thermos/cli/commands/test_import.py PRE-CREATION src/test/python/apache/thermos/cli/commands/test_simplerun.py PRE-CREATION src/test/python/apache/thermos/cli/test_common.py PRE-CREATION Diff: https://reviews.apache.org/r/31559/diff/ Testing --- ./pants test src/test/python/apache/thermos/cli:: Thanks, Brian Wickman
Re: Review Request 31559: Split thermos cli into pieces. Add custom aurora thermos entry point.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31559/ --- (Updated March 3, 2015, 10:14 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- Address joe's feedback Bugs: AURORA-1027 https://issues.apache.org/jira/browse/AURORA-1027 Repository: aurora Description --- This splits src/main/python/apache/thermos/bin/thermos.py into src/main/python/apache/thermos/cli/**. This also adds a path detector registry (AURORA-1027) for the client so that we can easily add a customized aurora entry point for thermos that is capable of finding checkpoint roots in executor sandboxes. Added said entry point in src/main/apache/aurora/tools/thermos.py -- open for other suggestions where this should live (possibly under aurora/executor/bin?) This is a drop-in replacement for the existing thermos binary. The cli/commands/*.py are mostly no-op splits with only a few lines changed in gc, status and tail to get_path_detector. Added a basic testing framework and an example test for simplerun but did not go with full coverage, instead opting to ship AURORA-1027 for now. Diffs (updated) - src/main/python/apache/aurora/tools/BUILD PRE-CREATION src/main/python/apache/aurora/tools/thermos.py PRE-CREATION src/main/python/apache/thermos/bin/BUILD 1a002da4f8294b3ecd43ab845eb15bc8ebd12b26 src/main/python/apache/thermos/bin/thermos.py 0853a9892399824385bee9e72db4c108f46fceda src/main/python/apache/thermos/cli/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/thermos.py PRE-CREATION src/main/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/main/python/apache/thermos/cli/commands/gc.py PRE-CREATION src/main/python/apache/thermos/cli/commands/help.py PRE-CREATION src/main/python/apache/thermos/cli/commands/inspect.py PRE-CREATION src/main/python/apache/thermos/cli/commands/kill.py PRE-CREATION src/main/python/apache/thermos/cli/commands/read.py PRE-CREATION src/main/python/apache/thermos/cli/commands/run.py PRE-CREATION src/main/python/apache/thermos/cli/commands/simplerun.py PRE-CREATION src/main/python/apache/thermos/cli/commands/status.py PRE-CREATION src/main/python/apache/thermos/cli/commands/tail.py PRE-CREATION src/main/python/apache/thermos/cli/common.py PRE-CREATION src/main/python/apache/thermos/cli/main.py PRE-CREATION src/test/python/apache/thermos/BUILD df50964851c6be89c646015edf91d07992743497 src/test/python/apache/thermos/bin/BUILD 596aeb200dc55b077ed73fc342b9187045732890 src/test/python/apache/thermos/bin/test_thermos.py 2d9d33397ae01b31ab011d958f5457311ef7ef20 src/test/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/test/python/apache/thermos/cli/commands/test_import.py PRE-CREATION src/test/python/apache/thermos/cli/commands/test_simplerun.py PRE-CREATION src/test/python/apache/thermos/cli/test_common.py PRE-CREATION Diff: https://reviews.apache.org/r/31559/diff/ Testing --- ./pants test src/test/python/apache/thermos/cli:: Thanks, Brian Wickman
Re: Review Request 31451: Port thermos observer to the path detector interface
On March 1, 2015, 8:47 p.m., Joe Smith wrote: src/main/python/apache/aurora/executor/common/resource_manager.py, line 96 https://reviews.apache.org/r/31451/diff/6/?file=880982#file880982line96 Why is this moving to kwargs? (My normal assumption is to stick with actual arguments) this is mostly so we don't have to duplicate all the options for TaskResourceMonitor. These options are really just around for mocking. On March 1, 2015, 8:47 p.m., Joe Smith wrote: src/main/python/apache/thermos/observer/detector.py, line 8 https://reviews.apache.org/r/31451/diff/6/?file=880992#file880992line8 You think something like this might help for a docstring? ``` class ObserverTaskDetector(object): The canonical way to watch (and be notified) of tasks transitioning state from active - finished - removed. ``` ObserverTaskDetector turns on-disk thermos task transitions into callback events. On March 1, 2015, 8:47 p.m., Joe Smith wrote: src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py, line 79 https://reviews.apache.org/r/31451/diff/6/?file=880998#file880998line79 Maybe add a TODO not to write to the filesystem? No. It is an integration test. On March 1, 2015, 8:47 p.m., Joe Smith wrote: src/test/python/apache/thermos/observer/test_detector.py, line 1 https://reviews.apache.org/r/31451/diff/6/?file=881004#file881004line1 add apache copywrite header? fixed On March 1, 2015, 8:47 p.m., Joe Smith wrote: src/test/python/apache/thermos/observer/test_detector.py, line 70 https://reviews.apache.org/r/31451/diff/6/?file=881004#file881004line70 putting this in a `setUp` method would remove the need for remembering `on_finished.reset_mock()` right? this would be true if each contextmanager block could live in its own test_... function, but because the state of the ObserverTaskDetector needs to be carried from context to context (e.g. active_tasks=TASK1, then finished_tasks=TASK1 to ensure that on_finished is called, etc) we can't tear down all test state, and instead need to reset the mocks. On March 1, 2015, 8:47 p.m., Joe Smith wrote: src/test/python/apache/thermos/observer/test_detector.py, line 76 https://reviews.apache.org/r/31451/diff/6/?file=881004#file881004line76 I think it's worthy to make this (and others) explicitly: assert on_active.mock_calls == [mock.call(TASK1[0], TASK1[1]] on_active.assert_called_once_with(X) is the same as assert on_active.mock_calls == [mock.call(X)], right? - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31451/#review74709 --- On Feb. 27, 2015, 11:36 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31451/ --- (Updated Feb. 27, 2015, 11:36 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Bugs: AURORA-1026 https://issues.apache.org/jira/browse/AURORA-1026 Repository: aurora Description --- This creates a new abstraction, the ObserverTaskDetector, which is responsible for managing state transitions for tasks for the observer. Adds some tests and better debug logging. Diffs - src/main/python/apache/aurora/executor/common/BUILD 8cef41d04aff9edc7da7053e05341274a9bd0834 src/main/python/apache/aurora/executor/common/resource_manager.py 08e02e41b581f275f070228bb23c4cf2a0489f9a src/main/python/apache/aurora/executor/thermos_task_runner.py 7a28e3255842e3e13a0866d6ad1bfc4cb64781e1 src/main/python/apache/thermos/bin/thermos.py 0853a9892399824385bee9e72db4c108f46fceda src/main/python/apache/thermos/common/path.py 846f507e2e097fc04fe0098a7250b40fefcfc6e2 src/main/python/apache/thermos/monitoring/disk.py 175ed3af6515e6107e297d91d4e30cbb3034faf7 src/main/python/apache/thermos/monitoring/monitor.py 11423bc1764c8380d8de4ad095c1e2748ebb78f8 src/main/python/apache/thermos/monitoring/resource.py b4cb881c87a09bb90a740f369a7a5fc5d75dbf04 src/main/python/apache/thermos/observer/BUILD ee65f3a46e1d339620e76cadae92c6678fc3510f src/main/python/apache/thermos/observer/bin/BUILD 15a03f74f204f58856f0843b9db05e83b89d1138 src/main/python/apache/thermos/observer/bin/thermos_observer.py effa8c19f963bf2792497f4a06049214ae30dfa5 src/main/python/apache/thermos/observer/detector.py PRE-CREATION src/main/python/apache/thermos/observer/http/file_browser.py 87ef9c8a29689c78a5e39a46cc53e4675e36a381 src/main/python/apache/thermos/observer/observed_task.py f33aecbc8f3c0a461ae3dba66fbd4986f544dc04 src/main/python/apache/thermos/observer
Re: Review Request 31451: Port thermos observer to the path detector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31451/ --- (Updated March 3, 2015, 10:58 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- Feedbackses Bugs: AURORA-1026 https://issues.apache.org/jira/browse/AURORA-1026 Repository: aurora Description --- This creates a new abstraction, the ObserverTaskDetector, which is responsible for managing state transitions for tasks for the observer. Adds some tests and better debug logging. Diffs (updated) - src/main/python/apache/aurora/executor/common/BUILD 8cef41d04aff9edc7da7053e05341274a9bd0834 src/main/python/apache/aurora/executor/common/resource_manager.py 08e02e41b581f275f070228bb23c4cf2a0489f9a src/main/python/apache/aurora/executor/thermos_task_runner.py 7a28e3255842e3e13a0866d6ad1bfc4cb64781e1 src/main/python/apache/thermos/bin/thermos.py 0853a9892399824385bee9e72db4c108f46fceda src/main/python/apache/thermos/common/path.py 846f507e2e097fc04fe0098a7250b40fefcfc6e2 src/main/python/apache/thermos/monitoring/disk.py 175ed3af6515e6107e297d91d4e30cbb3034faf7 src/main/python/apache/thermos/monitoring/monitor.py 11423bc1764c8380d8de4ad095c1e2748ebb78f8 src/main/python/apache/thermos/monitoring/resource.py b4cb881c87a09bb90a740f369a7a5fc5d75dbf04 src/main/python/apache/thermos/observer/BUILD ee65f3a46e1d339620e76cadae92c6678fc3510f src/main/python/apache/thermos/observer/bin/BUILD 15a03f74f204f58856f0843b9db05e83b89d1138 src/main/python/apache/thermos/observer/bin/thermos_observer.py effa8c19f963bf2792497f4a06049214ae30dfa5 src/main/python/apache/thermos/observer/detector.py PRE-CREATION src/main/python/apache/thermos/observer/http/file_browser.py 87ef9c8a29689c78a5e39a46cc53e4675e36a381 src/main/python/apache/thermos/observer/observed_task.py f33aecbc8f3c0a461ae3dba66fbd4986f544dc04 src/main/python/apache/thermos/observer/task_observer.py cd528dcca3f5a330359cf38005f3a1a0329a4886 src/test/python/apache/aurora/executor/BUILD b8dd28c3e64248d0181ad606d4c6de5d915ac706 src/test/python/apache/aurora/executor/common/BUILD 7b73f693d161cfd205435e4acb398f553b92389f src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py 8f288f6115ab52265dfada3f41d81271c55a src/test/python/apache/aurora/executor/test_thermos_executor.py 6cc928edc60d7a0f3b0e668edf68eb81b49cdb22 src/test/python/apache/thermos/bin/test_thermos.py 2d9d33397ae01b31ab011d958f5457311ef7ef20 src/test/python/apache/thermos/core/test_staged_kill.py faa23ae6afcaa443d23c65e5e027902961a0e52b src/test/python/apache/thermos/monitoring/test_resource.py 52d813946379bdc70c40ad079c74b54f60bd4b41 src/test/python/apache/thermos/observer/BUILD PRE-CREATION src/test/python/apache/thermos/observer/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/31451/diff/ Testing --- Manually launched observer and ran some thermos tasks. + mba=aurora=; ./pants test src/test/python/apache/thermos/observer/:: Thanks, Brian Wickman
Re: Review Request 31559: Split thermos cli into pieces. Add custom aurora thermos entry point.
On March 3, 2015, 10:31 p.m., Zameer Manji wrote: src/main/python/apache/thermos/cli/common.py, line 101 https://reviews.apache.org/r/31559/diff/6/?file=883670#file883670line101 shouldn't closing the checkpoint and killing the task runner be in a finally clause? No, it's ^C that should indicate whether or not we should kill the task. If the task exits normally, .close_ckpt/.kill will fail since the checkpoint is already closed and the task is in a terminal state. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31559/#review75063 --- On March 3, 2015, 10:14 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31559/ --- (Updated March 3, 2015, 10:14 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Bugs: AURORA-1027 https://issues.apache.org/jira/browse/AURORA-1027 Repository: aurora Description --- This splits src/main/python/apache/thermos/bin/thermos.py into src/main/python/apache/thermos/cli/**. This also adds a path detector registry (AURORA-1027) for the client so that we can easily add a customized aurora entry point for thermos that is capable of finding checkpoint roots in executor sandboxes. Added said entry point in src/main/apache/aurora/tools/thermos.py -- open for other suggestions where this should live (possibly under aurora/executor/bin?) This is a drop-in replacement for the existing thermos binary. The cli/commands/*.py are mostly no-op splits with only a few lines changed in gc, status and tail to get_path_detector. Added a basic testing framework and an example test for simplerun but did not go with full coverage, instead opting to ship AURORA-1027 for now. Diffs - src/main/python/apache/aurora/tools/BUILD PRE-CREATION src/main/python/apache/aurora/tools/thermos.py PRE-CREATION src/main/python/apache/thermos/bin/BUILD 1a002da4f8294b3ecd43ab845eb15bc8ebd12b26 src/main/python/apache/thermos/bin/thermos.py 0853a9892399824385bee9e72db4c108f46fceda src/main/python/apache/thermos/cli/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/thermos.py PRE-CREATION src/main/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/main/python/apache/thermos/cli/commands/gc.py PRE-CREATION src/main/python/apache/thermos/cli/commands/help.py PRE-CREATION src/main/python/apache/thermos/cli/commands/inspect.py PRE-CREATION src/main/python/apache/thermos/cli/commands/kill.py PRE-CREATION src/main/python/apache/thermos/cli/commands/read.py PRE-CREATION src/main/python/apache/thermos/cli/commands/run.py PRE-CREATION src/main/python/apache/thermos/cli/commands/simplerun.py PRE-CREATION src/main/python/apache/thermos/cli/commands/status.py PRE-CREATION src/main/python/apache/thermos/cli/commands/tail.py PRE-CREATION src/main/python/apache/thermos/cli/common.py PRE-CREATION src/main/python/apache/thermos/cli/main.py PRE-CREATION src/test/python/apache/thermos/BUILD df50964851c6be89c646015edf91d07992743497 src/test/python/apache/thermos/bin/BUILD 596aeb200dc55b077ed73fc342b9187045732890 src/test/python/apache/thermos/bin/test_thermos.py 2d9d33397ae01b31ab011d958f5457311ef7ef20 src/test/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/test/python/apache/thermos/cli/commands/test_import.py PRE-CREATION src/test/python/apache/thermos/cli/commands/test_simplerun.py PRE-CREATION src/test/python/apache/thermos/cli/test_common.py PRE-CREATION Diff: https://reviews.apache.org/r/31559/diff/ Testing --- ./pants test src/test/python/apache/thermos/cli:: Thanks, Brian Wickman
Re: Review Request 31559: Split thermos cli into pieces. Add custom aurora thermos entry point.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31559/ --- (Updated Feb. 27, 2015, 10:29 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- Address feedback and merge master. Bugs: AURORA-1027 https://issues.apache.org/jira/browse/AURORA-1027 Repository: aurora Description --- This splits src/main/python/apache/thermos/bin/thermos.py into src/main/python/apache/thermos/cli/**. This also adds a path detector registry (AURORA-1027) for the client so that we can easily add a customized aurora entry point for thermos that is capable of finding checkpoint roots in executor sandboxes. Added said entry point in src/main/apache/aurora/tools/thermos.py -- open for other suggestions where this should live (possibly under aurora/executor/bin?) This is a drop-in replacement for the existing thermos binary. The cli/commands/*.py are mostly no-op splits with only a few lines changed in gc, status and tail to get_path_detector. Added a basic testing framework and an example test for simplerun but did not go with full coverage, instead opting to ship AURORA-1027 for now. Diffs (updated) - src/main/python/apache/aurora/tools/BUILD PRE-CREATION src/main/python/apache/aurora/tools/thermos.py PRE-CREATION src/main/python/apache/thermos/bin/BUILD 1a002da4f8294b3ecd43ab845eb15bc8ebd12b26 src/main/python/apache/thermos/bin/thermos.py 0853a9892399824385bee9e72db4c108f46fceda src/main/python/apache/thermos/cli/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/thermos.py PRE-CREATION src/main/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/main/python/apache/thermos/cli/commands/gc.py PRE-CREATION src/main/python/apache/thermos/cli/commands/help.py PRE-CREATION src/main/python/apache/thermos/cli/commands/inspect.py PRE-CREATION src/main/python/apache/thermos/cli/commands/kill.py PRE-CREATION src/main/python/apache/thermos/cli/commands/read.py PRE-CREATION src/main/python/apache/thermos/cli/commands/run.py PRE-CREATION src/main/python/apache/thermos/cli/commands/simplerun.py PRE-CREATION src/main/python/apache/thermos/cli/commands/status.py PRE-CREATION src/main/python/apache/thermos/cli/commands/tail.py PRE-CREATION src/main/python/apache/thermos/cli/common.py PRE-CREATION src/main/python/apache/thermos/cli/main.py PRE-CREATION src/test/python/apache/thermos/BUILD df50964851c6be89c646015edf91d07992743497 src/test/python/apache/thermos/bin/BUILD 596aeb200dc55b077ed73fc342b9187045732890 src/test/python/apache/thermos/bin/test_thermos.py 2d9d33397ae01b31ab011d958f5457311ef7ef20 src/test/python/apache/thermos/cli/BUILD PRE-CREATION src/test/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/test/python/apache/thermos/cli/commands/test_import.py PRE-CREATION src/test/python/apache/thermos/cli/commands/test_simplerun.py PRE-CREATION src/test/python/apache/thermos/cli/test_common.py PRE-CREATION src/test/python/apache/thermos/cli/test_main.py PRE-CREATION Diff: https://reviews.apache.org/r/31559/diff/ Testing --- ./pants test src/test/python/apache/thermos/cli:: Thanks, Brian Wickman
Re: Review Request 31559: Split thermos cli into pieces. Add custom aurora thermos entry point.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31559/ --- (Updated Feb. 27, 2015, 10:48 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- Add a spec_set on the options_mock for cli/commands/test_simplerun. Bugs: AURORA-1027 https://issues.apache.org/jira/browse/AURORA-1027 Repository: aurora Description --- This splits src/main/python/apache/thermos/bin/thermos.py into src/main/python/apache/thermos/cli/**. This also adds a path detector registry (AURORA-1027) for the client so that we can easily add a customized aurora entry point for thermos that is capable of finding checkpoint roots in executor sandboxes. Added said entry point in src/main/apache/aurora/tools/thermos.py -- open for other suggestions where this should live (possibly under aurora/executor/bin?) This is a drop-in replacement for the existing thermos binary. The cli/commands/*.py are mostly no-op splits with only a few lines changed in gc, status and tail to get_path_detector. Added a basic testing framework and an example test for simplerun but did not go with full coverage, instead opting to ship AURORA-1027 for now. Diffs (updated) - src/main/python/apache/aurora/tools/BUILD PRE-CREATION src/main/python/apache/aurora/tools/thermos.py PRE-CREATION src/main/python/apache/thermos/bin/BUILD 1a002da4f8294b3ecd43ab845eb15bc8ebd12b26 src/main/python/apache/thermos/bin/thermos.py 0853a9892399824385bee9e72db4c108f46fceda src/main/python/apache/thermos/cli/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/thermos.py PRE-CREATION src/main/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/main/python/apache/thermos/cli/commands/gc.py PRE-CREATION src/main/python/apache/thermos/cli/commands/help.py PRE-CREATION src/main/python/apache/thermos/cli/commands/inspect.py PRE-CREATION src/main/python/apache/thermos/cli/commands/kill.py PRE-CREATION src/main/python/apache/thermos/cli/commands/read.py PRE-CREATION src/main/python/apache/thermos/cli/commands/run.py PRE-CREATION src/main/python/apache/thermos/cli/commands/simplerun.py PRE-CREATION src/main/python/apache/thermos/cli/commands/status.py PRE-CREATION src/main/python/apache/thermos/cli/commands/tail.py PRE-CREATION src/main/python/apache/thermos/cli/common.py PRE-CREATION src/main/python/apache/thermos/cli/main.py PRE-CREATION src/test/python/apache/thermos/BUILD df50964851c6be89c646015edf91d07992743497 src/test/python/apache/thermos/bin/BUILD 596aeb200dc55b077ed73fc342b9187045732890 src/test/python/apache/thermos/bin/test_thermos.py 2d9d33397ae01b31ab011d958f5457311ef7ef20 src/test/python/apache/thermos/cli/BUILD PRE-CREATION src/test/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/test/python/apache/thermos/cli/commands/test_import.py PRE-CREATION src/test/python/apache/thermos/cli/commands/test_simplerun.py PRE-CREATION src/test/python/apache/thermos/cli/test_common.py PRE-CREATION src/test/python/apache/thermos/cli/test_main.py PRE-CREATION Diff: https://reviews.apache.org/r/31559/diff/ Testing --- ./pants test src/test/python/apache/thermos/cli:: Thanks, Brian Wickman
Re: Review Request 31559: Split thermos cli into pieces. Add custom aurora thermos entry point.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31559/ --- (Updated Feb. 27, 2015, 11:37 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- Merge master build fix. Bugs: AURORA-1027 https://issues.apache.org/jira/browse/AURORA-1027 Repository: aurora Description --- This splits src/main/python/apache/thermos/bin/thermos.py into src/main/python/apache/thermos/cli/**. This also adds a path detector registry (AURORA-1027) for the client so that we can easily add a customized aurora entry point for thermos that is capable of finding checkpoint roots in executor sandboxes. Added said entry point in src/main/apache/aurora/tools/thermos.py -- open for other suggestions where this should live (possibly under aurora/executor/bin?) This is a drop-in replacement for the existing thermos binary. The cli/commands/*.py are mostly no-op splits with only a few lines changed in gc, status and tail to get_path_detector. Added a basic testing framework and an example test for simplerun but did not go with full coverage, instead opting to ship AURORA-1027 for now. Diffs (updated) - src/main/python/apache/aurora/tools/BUILD PRE-CREATION src/main/python/apache/aurora/tools/thermos.py PRE-CREATION src/main/python/apache/thermos/bin/BUILD 1a002da4f8294b3ecd43ab845eb15bc8ebd12b26 src/main/python/apache/thermos/bin/thermos.py 0853a9892399824385bee9e72db4c108f46fceda src/main/python/apache/thermos/cli/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/BUILD PRE-CREATION src/main/python/apache/thermos/cli/bin/thermos.py PRE-CREATION src/main/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/main/python/apache/thermos/cli/commands/gc.py PRE-CREATION src/main/python/apache/thermos/cli/commands/help.py PRE-CREATION src/main/python/apache/thermos/cli/commands/inspect.py PRE-CREATION src/main/python/apache/thermos/cli/commands/kill.py PRE-CREATION src/main/python/apache/thermos/cli/commands/read.py PRE-CREATION src/main/python/apache/thermos/cli/commands/run.py PRE-CREATION src/main/python/apache/thermos/cli/commands/simplerun.py PRE-CREATION src/main/python/apache/thermos/cli/commands/status.py PRE-CREATION src/main/python/apache/thermos/cli/commands/tail.py PRE-CREATION src/main/python/apache/thermos/cli/common.py PRE-CREATION src/main/python/apache/thermos/cli/main.py PRE-CREATION src/test/python/apache/thermos/BUILD df50964851c6be89c646015edf91d07992743497 src/test/python/apache/thermos/bin/BUILD 596aeb200dc55b077ed73fc342b9187045732890 src/test/python/apache/thermos/bin/test_thermos.py 2d9d33397ae01b31ab011d958f5457311ef7ef20 src/test/python/apache/thermos/cli/BUILD PRE-CREATION src/test/python/apache/thermos/cli/commands/BUILD PRE-CREATION src/test/python/apache/thermos/cli/commands/test_import.py PRE-CREATION src/test/python/apache/thermos/cli/commands/test_simplerun.py PRE-CREATION src/test/python/apache/thermos/cli/test_common.py PRE-CREATION src/test/python/apache/thermos/cli/test_main.py PRE-CREATION Diff: https://reviews.apache.org/r/31559/diff/ Testing --- ./pants test src/test/python/apache/thermos/cli:: Thanks, Brian Wickman
Re: Review Request 31451: Port thermos observer to the path detector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31451/ --- (Updated Feb. 26, 2015, 7:46 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- Remove tbds Bugs: AURORA-1026 https://issues.apache.org/jira/browse/AURORA-1026 Repository: aurora Description --- This creates a new abstraction, the ObserverTaskDetector, which is responsible for managing state transitions for tasks for the observer. Adds some tests and better debug logging. Diffs (updated) - src/main/python/apache/aurora/executor/common/resource_manager.py 08e02e41b581f275f070228bb23c4cf2a0489f9a src/main/python/apache/aurora/executor/thermos_task_runner.py 7a28e3255842e3e13a0866d6ad1bfc4cb64781e1 src/main/python/apache/thermos/bin/thermos.py 0853a9892399824385bee9e72db4c108f46fceda src/main/python/apache/thermos/common/path.py 846f507e2e097fc04fe0098a7250b40fefcfc6e2 src/main/python/apache/thermos/monitoring/disk.py 175ed3af6515e6107e297d91d4e30cbb3034faf7 src/main/python/apache/thermos/monitoring/monitor.py 11423bc1764c8380d8de4ad095c1e2748ebb78f8 src/main/python/apache/thermos/monitoring/resource.py b4cb881c87a09bb90a740f369a7a5fc5d75dbf04 src/main/python/apache/thermos/observer/BUILD ee65f3a46e1d339620e76cadae92c6678fc3510f src/main/python/apache/thermos/observer/bin/BUILD 15a03f74f204f58856f0843b9db05e83b89d1138 src/main/python/apache/thermos/observer/bin/thermos_observer.py effa8c19f963bf2792497f4a06049214ae30dfa5 src/main/python/apache/thermos/observer/detector.py PRE-CREATION src/main/python/apache/thermos/observer/http/file_browser.py 87ef9c8a29689c78a5e39a46cc53e4675e36a381 src/main/python/apache/thermos/observer/observed_task.py f33aecbc8f3c0a461ae3dba66fbd4986f544dc04 src/main/python/apache/thermos/observer/task_observer.py cd528dcca3f5a330359cf38005f3a1a0329a4886 src/test/python/apache/thermos/observer/BUILD PRE-CREATION src/test/python/apache/thermos/observer/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/31451/diff/ Testing --- Manually launched observer and ran some thermos tasks. + mba=aurora=; ./pants test src/test/python/apache/thermos/observer/:: Thanks, Brian Wickman
Re: Review Request 31491: Add basic test coverage to the TaskGarbageCollector.
On Feb. 26, 2015, 7:46 p.m., Joshua Cohen wrote: src/main/python/apache/thermos/monitoring/garbage.py, line 83 https://reviews.apache.org/r/31491/diff/1/?file=878599#file878599line83 What's the purpose of these comments? ugh, keep forgetting to delete these comments On Feb. 26, 2015, 7:46 p.m., Joshua Cohen wrote: src/main/python/apache/thermos/monitoring/garbage.py, lines 44-47 https://reviews.apache.org/r/31491/diff/1/?file=878599#file878599line44 safe_mtime? derp yeah - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31491/#review74343 --- On Feb. 26, 2015, 7:20 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31491/ --- (Updated Feb. 26, 2015, 7:20 p.m.) Review request for Aurora, Joshua Cohen and Stephan Erb. Bugs: AURORA-1155 https://issues.apache.org/jira/browse/AURORA-1155 Repository: aurora Description --- Add basic test coverage to the TaskGarbageCollector. Diffs - src/main/python/apache/thermos/monitoring/garbage.py 53bb0345f963ae168206f34af1d82c67427f41fe src/test/python/apache/thermos/monitoring/BUILD 9c267775080e0bde08ff4e9b01862c6bee187634 src/test/python/apache/thermos/monitoring/test_garbage.py PRE-CREATION Diff: https://reviews.apache.org/r/31491/diff/ Testing --- ./pants test src/test/python/apache/thermos/monitoring:test_garbage Thanks, Brian Wickman
Re: Review Request 31491: Add basic test coverage to the TaskGarbageCollector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31491/#review74381 --- src/main/python/apache/thermos/monitoring/garbage.py https://reviews.apache.org/r/31491/#comment120949 Thanks for the catch. - Brian Wickman On Feb. 26, 2015, 8 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31491/ --- (Updated Feb. 26, 2015, 8 p.m.) Review request for Aurora, Joshua Cohen and Stephan Erb. Bugs: AURORA-1155 https://issues.apache.org/jira/browse/AURORA-1155 Repository: aurora Description --- Add basic test coverage to the TaskGarbageCollector. Diffs - src/main/python/apache/thermos/monitoring/garbage.py 53bb0345f963ae168206f34af1d82c67427f41fe src/test/python/apache/thermos/monitoring/BUILD 9c267775080e0bde08ff4e9b01862c6bee187634 src/test/python/apache/thermos/monitoring/test_garbage.py PRE-CREATION Diff: https://reviews.apache.org/r/31491/diff/ Testing --- ./pants test src/test/python/apache/thermos/monitoring:test_garbage Thanks, Brian Wickman
Re: Review Request 31251: Fix swallowed exceptions in health check test.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31251/ --- (Updated Feb. 23, 2015, 8:23 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- s/log.warning/log.error/ Repository: aurora Description --- Fix swallowed exceptions in health check test. One of the health checkers was raising uncaught exceptions on a separate thread. The Python interpreter on CI *might* behave slightly differently and fail here. To be honest I've not been able to reproduce the CI behavior, so hopefully buildbot can chime in here. Diffs (updated) - src/main/python/apache/aurora/executor/common/health_checker.py 0d3365d395c65793b90ce96f277b7f4e4b1dcb99 src/test/python/apache/aurora/executor/common/test_health_checker.py 4e09d309d402cf678c69d033e5219e9c8e4ccaaa src/test/python/apache/aurora/executor/test_gc_executor.py 27dee7fa10a4141ec7e9f4440bde2dd257db1cc6 Diff: https://reviews.apache.org/r/31251/diff/ Testing --- Ran the following about 100 times: THERMOS_DEBUG=1 ./pants test.pytest --no-fast --options='-vs' src/test/python/apache/aurora/executor/common:health_checker Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review73337 --- @Reviewbot retry - Brian Wickman On Feb. 20, 2015, 7:06 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 20, 2015, 7:06 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 6530f4914736754f92ba192c1a345e4b7e4a5398 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py dbec82ffe4e155059f3e6b1aa0e67ec4c52a9611 src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py 27dee7fa10a4141ec7e9f4440bde2dd257db1cc6 Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30647: Instrument the HealthChecker to export stats.
On Feb. 20, 2015, 10:30 p.m., Bill Farner wrote: src/test/python/apache/aurora/executor/common/test_health_checker.py, line 136 https://reviews.apache.org/r/30647/diff/8/?file=870602#file870602line136 Is it necessary to use a real clock + real threads + sleeps in this test? I'm not confident i can confirm that this test will not be flaky with these in place. this is not a real clock. it's a fake clock that allows us to test threaded behavior: https://github.com/twitter/commons/blob/master/src/python/twitter/common/testing/clock.py On Feb. 20, 2015, 10:30 p.m., Bill Farner wrote: src/main/python/apache/aurora/executor/common/health_checker.py, line 161 https://reviews.apache.org/r/30647/diff/8/?file=870600#file870600line161 nit - can you include the unit in the name? i've seen time-based things misused more often than i care to recall sure - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/#review73324 --- On Feb. 20, 2015, 7:33 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- (Updated Feb. 20, 2015, 7:33 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Bugs: AURORA-1062 https://issues.apache.org/jira/browse/AURORA-1062 Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py a4e215d4422e3ada7b7913eaab105fdf030695c5 src/test/python/apache/aurora/executor/test_thermos_executor.py c8fab307d17949a8157659c4b3944ec7520feb9d Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 31251: Fix swallowed exceptions in health check test.
(maximum_executor_lifetime=Amount(500, Time.MILLISECONDS)) as ( proxy_driver, executor): executor._clock.tick(1) proxy_driver.stopped.wait(timeout=1.0) assert proxy_driver.stopped.is_set() E assert bound method _Event.isSet of threading._Event object at 0x1069f0090() E+ where bound method _Event.isSet of threading._Event object at 0x1069f0090 = threading._Event object at 0x1069f0090.is_set E+where threading._Event object at 0x1069f0090 = test_gc_executor.ProxyDriver object at 0x1069f0350.stopped src/test/python/apache/aurora/executor/test_gc_executor.py:509: AssertionError == 1 failed, 27 passed in 11.85 seconds == src.test.python.apache.aurora.executor.gc_executor . FAILURE FAILURE - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31251/#review73380 --- On Feb. 21, 2015, 1:09 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31251/ --- (Updated Feb. 21, 2015, 1:09 a.m.) Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- Fix swallowed exceptions in health check test. One of the health checkers was raising uncaught exceptions on a separate thread. The Python interpreter on CI *might* behave slightly differently and fail here. To be honest I've not been able to reproduce the CI behavior, so hopefully buildbot can chime in here. Diffs - src/main/python/apache/aurora/executor/common/health_checker.py 0d3365d395c65793b90ce96f277b7f4e4b1dcb99 src/test/python/apache/aurora/executor/common/test_health_checker.py 4e09d309d402cf678c69d033e5219e9c8e4ccaaa Diff: https://reviews.apache.org/r/31251/diff/ Testing --- Ran the following about 100 times: THERMOS_DEBUG=1 ./pants test.pytest --no-fast --options='-vs' src/test/python/apache/aurora/executor/common:health_checker Thanks, Brian Wickman
Review Request 31251: Fix swallowed exceptions in health check test.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31251/ --- Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- Fix swallowed exceptions in health check test. One of the health checkers was raising uncaught exceptions on a separate thread. The Python interpreter on CI *might* behave slightly differently and fail here. To be honest I've not been able to reproduce the CI behavior, so hopefully buildbot can chime in here. Diffs - src/main/python/apache/aurora/executor/common/health_checker.py 0d3365d395c65793b90ce96f277b7f4e4b1dcb99 src/test/python/apache/aurora/executor/common/test_health_checker.py 4e09d309d402cf678c69d033e5219e9c8e4ccaaa Diff: https://reviews.apache.org/r/31251/diff/ Testing --- Ran the following about 100 times: THERMOS_DEBUG=1 ./pants test.pytest --no-fast --options='-vs' src/test/python/apache/aurora/executor/common:health_checker Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
On Feb. 10, 2015, 7:12 p.m., Joshua Cohen wrote: src/main/python/apache/thermos/monitoring/garbage.py, line 120 https://reviews.apache.org/r/30749/diff/9/?file=859049#file859049line120 Are we no longer worried about the scenario where the checkpoint root is $HOME or /? The checkpoint root could be set to this but only maliciously. On Feb. 10, 2015, 7:12 p.m., Joshua Cohen wrote: src/test/python/apache/aurora/executor/test_gc_executor.py, line 180 https://reviews.apache.org/r/30749/diff/9/?file=859053#file859053line180 s/'fake_root'/FAKE_ROOT fixed On Feb. 10, 2015, 7:12 p.m., Joshua Cohen wrote: src/test/python/apache/aurora/executor/test_gc_executor.py, lines 210-211 https://reviews.apache.org/r/30749/diff/9/?file=859053#file859053line210 Am I missing something or is this a duplicate of the make_task defined above? totally a dupe, deleted. On Feb. 10, 2015, 7:12 p.m., Joshua Cohen wrote: src/test/python/apache/aurora/executor/test_gc_executor.py, line 254 https://reviews.apache.org/r/30749/diff/9/?file=859053#file859053line254 nit: maybe give task_id a default value in make_task instead of having to always pass in the same placeholder value? Nah, because in all of these assertions we actually need to know the task id so it wouldn't make sense to omit from make_task(). - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71833 --- On Feb. 20, 2015, 7:06 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 20, 2015, 7:06 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 6530f4914736754f92ba192c1a345e4b7e4a5398 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py dbec82ffe4e155059f3e6b1aa0e67ec4c52a9611 src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py 27dee7fa10a4141ec7e9f4440bde2dd257db1cc6 Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
On Feb. 10, 2015, 7:19 p.m., Joe Smith wrote: src/main/python/apache/aurora/executor/gc_executor.py, line 73 https://reviews.apache.org/r/30749/diff/9/?file=859047#file859047line73 Some documentation would go a long way here. I assume this is the way to go from `task_id` - `path_of_the_checkpoint_stream_on_disk` ? Added some commentary. On Feb. 10, 2015, 7:19 p.m., Joe Smith wrote: src/main/python/apache/aurora/executor/gc_executor.py, line 129 https://reviews.apache.org/r/30749/diff/9/?file=859047#file859047line129 s/PathDetector/path_detector, I think whoops, thanks. On Feb. 10, 2015, 7:19 p.m., Joe Smith wrote: src/main/python/apache/aurora/executor/gc_executor.py, line 135 https://reviews.apache.org/r/30749/diff/9/?file=859047#file859047line135 ``` :param task: an instance of a task to retrieve checkpoint path :type task: RootedTask instance ``` Does it make sense to pydoc a private method? i'll add it but i also don't want to necessarily set a precedent. On Feb. 10, 2015, 7:19 p.m., Joe Smith wrote: src/main/python/apache/thermos/monitoring/garbage.py, line 31 https://reviews.apache.org/r/30749/diff/9/?file=859049#file859049line31 Mind adding a docstring here and for the `__init__` ? done On Feb. 10, 2015, 7:19 p.m., Joe Smith wrote: src/main/python/apache/thermos/monitoring/garbage.py, line 173 https://reviews.apache.org/r/30749/diff/9/?file=859049#file859049line173 Everytime I see a namedtuple I cringe a bit (since I really only see them as a poor replacement for spec'd mocks) but if you don't mind adding docs here that'd be ~alright. This code is on the chopping block, so I wouldn't lament the general practices here too much. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71831 --- On Feb. 9, 2015, 11:16 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 11:16 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
On Feb. 20, 2015, 11:11 p.m., Brian Wickman wrote: @Reviewbot retry Bill Farner wrote: Is that test known to be flaky? If so, can you file a ticket? https://reviews.apache.org/r/30647/ should deflake it. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review73337 --- On Feb. 20, 2015, 7:06 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 20, 2015, 7:06 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 6530f4914736754f92ba192c1a345e4b7e4a5398 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py dbec82ffe4e155059f3e6b1aa0e67ec4c52a9611 src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py 27dee7fa10a4141ec7e9f4440bde2dd257db1cc6 Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30647: Instrument the HealthChecker to export stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- (Updated Feb. 17, 2015, 10:46 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Changes --- Added dependency on r/31137 Bugs: AURORA-1062 https://issues.apache.org/jira/browse/AURORA-1062 Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 src/test/python/apache/aurora/executor/test_thermos_executor.py c8fab307d17949a8157659c4b3944ec7520feb9d Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 31138: Add ability to pass configurable options to pytest.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31138/#review72840 --- Ship it! Ship It! - Brian Wickman On Feb. 17, 2015, 11:12 p.m., Joshua Cohen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31138/ --- (Updated Feb. 17, 2015, 11:12 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Repository: aurora Description --- Update jenkins build script to run python tests in verbose mode. This will let us more easily determine when a test causes a build to hang. Diffs - build-support/jenkins/build.sh 6f9f017f00b49174dbd5b3f70a4923d89a5f51a1 Diff: https://reviews.apache.org/r/31138/diff/ Testing --- Ran build.sh Thanks, Joshua Cohen
Re: Review Request 31124: Fix build break - remove unused imports.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31124/#review72752 --- Ship it! Ship It! - Brian Wickman On Feb. 17, 2015, 6:09 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31124/ --- (Updated Feb. 17, 2015, 6:09 p.m.) Review request for Aurora, Brian Wickman and Zameer Manji. Repository: aurora Description --- Addresses these style check errors: ``` F401:ERROR src/main/python/apache/aurora/executor/executor_vars.py:017 'BadZipfile' imported but unused |from zipfile import BadZipfile F401:ERROR src/main/python/apache/aurora/executor/executor_vars.py:020 'PexInfo' imported but unused |from pex.pex import PexInfo F401:ERROR src/main/python/apache/aurora/executor/executor_vars.py:022 'NamedGauge' imported but unused |from twitter.common.metrics import LambdaGauge, MutatorGauge, NamedGauge, Observable F401:ERROR src/main/python/apache/aurora/executor/executor_vars.py:024 'ScanfParser' imported but unused |from twitter.common.string.scanf import ScanfParser ``` Diffs - src/main/python/apache/aurora/executor/executor_vars.py 283b62459a2a4173a96661129eb259b2384c23ca Diff: https://reviews.apache.org/r/31124/diff/ Testing --- Thanks, Bill Farner
Review Request 31137: Update aurora to commons 0.3.3, unflake tests using ThreadedClock
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31137/ --- Review request for Aurora, Joshua Cohen and Joe Smith. Repository: aurora Description --- SSIA. One seemingly incorrect test was removed. It was unclear to me what behavior it was testing exactly, and it did not consistently pass. Diffs - 3rdparty/python/requirements.txt 9f7bea9c0dce620c436332e1f4f4c9e3df7f66f5 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/31137/diff/ Testing --- ./pants test.pytest --no-fast src/test/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 6:45 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Revert unintended pants.ini change. Pull in a slice of r/30768 to pass e2e. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/aurora/executor/thermos_task_runner.py 9ff8c5379aad7ac05e44b1f5a2b74f759f26 src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30768: Reject None values for TaskPath
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/#review71649 --- Ship it! Ship It! - Brian Wickman On Feb. 8, 2015, 9:34 p.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30768/ --- (Updated Feb. 8, 2015, 9:34 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-1115 https://issues.apache.org/jira/browse/AURORA-1115 Repository: aurora Description --- This patch modifies TaskPath to reject None values. Diffs - src/main/python/apache/aurora/executor/thermos_task_runner.py 7b346e253677ee9b42c57782f7f67ff63b6a0083 src/main/python/apache/thermos/common/path.py 9e617051f16f4270b3958f48e0cc43706d245eec src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py PRE-CREATION src/test/python/apache/thermos/common/test_pathspec.py 3437b196d33d7c2ff6ba292ff99b6881954e7ecb src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 989801cfcbd19109ac140b01cd3024d70c78c829 Diff: https://reviews.apache.org/r/30768/diff/ Testing --- ./pants test src/test/python/apache/thermos:: ./pants test src/test/python/apache/aurora/executor:: ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Zameer Manji
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 6:56 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Post master merge. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
On Feb. 8, 2015, 2:20 a.m., Zameer Manji wrote: src/main/python/apache/aurora/executor/gc_executor.py, line 73 https://reviews.apache.org/r/30749/diff/4/?file=853457#file853457line73 What's the benefit of using a namedtuple here? As opposed to? This means we don't have to change every function signature in this file. (I suppose on the other hand we get less type safety because functions called w/o the correct number of parameters would at least TypeError.) On Feb. 8, 2015, 2:20 a.m., Zameer Manji wrote: src/main/python/apache/aurora/executor/gc_executor.py, line 411 https://reviews.apache.org/r/30749/diff/4/?file=853457#file853457line411 Shouldn't this log messsage be inside the collector implementation? The gc executor is on its deathbed -- I propose not changing anything not completely necessary for its deprecation. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71567 --- On Feb. 7, 2015, 2:27 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 7, 2015, 2:27 a.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - pants.ini 99648e46ca755c3c8e22d90c7b682107d8dee333 src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 8:24 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Argh, can't catch a break. Fixes build typo. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 8:22 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Adds entry point tests for gc/thermos executors, since I got some of the import wiring wrong: vagrant@192:/var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/latest$ cat stderr WARNING: Logging before InitGoogleLogging() is written to STDERR I0209 19:16:58.301468 22286 fetcher.cpp:76] Fetching URI '/home/vagrant/aurora/dist/gc_executor.pex' I0209 19:16:58.301730 22286 fetcher.cpp:179] Copying resource from '/home/vagrant/aurora/dist/gc_executor.pex' to '/var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/6f1f2749-66a9-41e7-bb68-5e31c08cac6d' Traceback (most recent call last): File /var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/6f1f2749-66a9-41e7-bb68-5e31c08cac6d/gc_executor.pex/.bootstrap/_pex/pex.py, line 272, in execute self.execute_entry(entry_point, args) File /var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/6f1f2749-66a9-41e7-bb68-5e31c08cac6d/gc_executor.pex/.bootstrap/_pex/pex.py, line 320, in execute_entry runner(entry_point) File /var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/6f1f2749-66a9-41e7-bb68-5e31c08cac6d/gc_executor.pex/.bootstrap/_pex/pex.py, line 342, in execute_pkg_resources runner = entry.load(require=False) # trust that the environment is sane File /var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/6f1f2749-66a9-41e7-bb68-5e31c08cac6d/gc_executor.pex/.bootstrap/pkg_resources.py, line 2048, in load entry = __import__(self.module_name, globals(),globals(), ['__name__']) File /var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/6f1f2749-66a9-41e7-bb68-5e31c08cac6d/gc_executor.pex/apache/aurora/executor/bin/gc_executor_main.py, line 27, in module ImportError: No module named executor_detector vagrant@192:/var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/latest$ Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30647: Instrument the HealthChecker to export stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/#review71711 --- ping. - Brian Wickman On Feb. 6, 2015, 11:13 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- (Updated Feb. 6, 2015, 11:13 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Bugs: AURORA-1062 https://issues.apache.org/jira/browse/AURORA-1062 Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 src/test/python/apache/aurora/executor/test_thermos_executor.py c8fab307d17949a8157659c4b3944ec7520feb9d Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 11:16 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Checkstyle Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 27182: Add a test for the thermos resource module
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/#review71520 --- src/test/python/apache/thermos/monitoring/test_resource.py https://reviews.apache.org/r/27182/#comment117267 these aren't explicitly covered by your BUILD target src/test/python/apache/thermos/monitoring/test_resource.py https://reviews.apache.org/r/27182/#comment117268 ditto re: coverage - Brian Wickman On Feb. 6, 2015, 10 p.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/ --- (Updated Feb. 6, 2015, 10 p.m.) Review request for Aurora and Brian Wickman. Repository: aurora Description --- Add a test for the thermos resource module Diffs - src/main/python/apache/thermos/monitoring/monitor.py 8f87f5ffc39c87e87ff78b941ea30df7138bd1ef src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_resource.py PRE-CREATION Diff: https://reviews.apache.org/r/27182/diff/ Testing --- [tw-mbp-jsmith incubator-aurora (yasumoto/thermos_test)]$ ./pants test.pytest ./src/test/python/apache/thermos/monitoring:test_resource 21:15:56 00:00 [main] (To run a reporting server: ./pants server) 21:15:56 00:00 [bootstrap] 21:15:57 00:01 [setup] 21:15:57 00:01 [parse] Executing tasks in goals: bootstrap - imports - gen - check-exclusives - resolve - compile - resources - test 21:15:57 00:01 [bootstrap] 21:15:57 00:01 [bootstrap-jvm-tools] 21:15:57 00:01 [imports] 21:15:57 00:01 [ivy-imports] 21:15:57 00:01 [gen] 21:15:57 00:01 [thrift] 21:15:57 00:01 [scrooge] 21:15:57 00:01 [protoc] 21:15:57 00:01 [antlr] 21:15:57 00:01 [ragel] 21:15:57 00:01 [jaxb] 21:15:57 00:01 [wire] 21:15:57 00:01 [aapt] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [resolve] 21:15:57 00:01 [ivy] 21:15:57 00:01 [compile] 21:15:57 00:01 [jvm] 21:15:57 00:01 [jvm-compilers] 21:15:57 00:01 [resources] 21:15:57 00:01 [prepare] 21:15:57 00:01 [test] 21:15:57 00:01 [run_prep_command] 21:15:57 00:01 [pytest] 21:15:57 00:01 [run] == test session starts === platform darwin -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4 plugins: cov, timeout collected 5 items src/test/python/apache/thermos/monitoring/test_resource.py . 5 passed in 0.18 seconds 21:16:04 00:08 [junit] 21:16:04 00:08 [specs] SUCCESS Thanks, Joe Smith
Re: Review Request 27182: Add a test for the thermos resource module
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/#review71523 --- Tried to merge but got a conflict. Could you re-merge this with master and re-publish so that I can patch to master? - Brian Wickman On Feb. 6, 2015, 10 p.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/ --- (Updated Feb. 6, 2015, 10 p.m.) Review request for Aurora and Brian Wickman. Repository: aurora Description --- Add a test for the thermos resource module Diffs - src/main/python/apache/thermos/monitoring/monitor.py 8f87f5ffc39c87e87ff78b941ea30df7138bd1ef src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_resource.py PRE-CREATION Diff: https://reviews.apache.org/r/27182/diff/ Testing --- [tw-mbp-jsmith incubator-aurora (yasumoto/thermos_test)]$ ./pants test.pytest ./src/test/python/apache/thermos/monitoring:test_resource 21:15:56 00:00 [main] (To run a reporting server: ./pants server) 21:15:56 00:00 [bootstrap] 21:15:57 00:01 [setup] 21:15:57 00:01 [parse] Executing tasks in goals: bootstrap - imports - gen - check-exclusives - resolve - compile - resources - test 21:15:57 00:01 [bootstrap] 21:15:57 00:01 [bootstrap-jvm-tools] 21:15:57 00:01 [imports] 21:15:57 00:01 [ivy-imports] 21:15:57 00:01 [gen] 21:15:57 00:01 [thrift] 21:15:57 00:01 [scrooge] 21:15:57 00:01 [protoc] 21:15:57 00:01 [antlr] 21:15:57 00:01 [ragel] 21:15:57 00:01 [jaxb] 21:15:57 00:01 [wire] 21:15:57 00:01 [aapt] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [resolve] 21:15:57 00:01 [ivy] 21:15:57 00:01 [compile] 21:15:57 00:01 [jvm] 21:15:57 00:01 [jvm-compilers] 21:15:57 00:01 [resources] 21:15:57 00:01 [prepare] 21:15:57 00:01 [test] 21:15:57 00:01 [run_prep_command] 21:15:57 00:01 [pytest] 21:15:57 00:01 [run] == test session starts === platform darwin -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4 plugins: cov, timeout collected 5 items src/test/python/apache/thermos/monitoring/test_resource.py . 5 passed in 0.18 seconds 21:16:04 00:08 [junit] 21:16:04 00:08 [specs] SUCCESS Thanks, Joe Smith
Review Request 30752: Fix executor builds.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30752/ --- Review request for Aurora. Repository: aurora Description --- The executor binary targets were broken when I moved the executor_detector from apache.aurora.executor to apache.aurora.executor.common Diffs - src/main/python/apache/aurora/executor/bin/BUILD 4d4f3ab9442ceda912ac4ce07eb15e291a4ed08c Diff: https://reviews.apache.org/r/30752/diff/ Testing --- Built the binaries. Thanks, Brian Wickman
Re: Review Request 27182: Add a test for the thermos resource module
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/#review71522 --- Ship it! Ship It! - Brian Wickman On Feb. 6, 2015, 10 p.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/ --- (Updated Feb. 6, 2015, 10 p.m.) Review request for Aurora and Brian Wickman. Repository: aurora Description --- Add a test for the thermos resource module Diffs - src/main/python/apache/thermos/monitoring/monitor.py 8f87f5ffc39c87e87ff78b941ea30df7138bd1ef src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_resource.py PRE-CREATION Diff: https://reviews.apache.org/r/27182/diff/ Testing --- [tw-mbp-jsmith incubator-aurora (yasumoto/thermos_test)]$ ./pants test.pytest ./src/test/python/apache/thermos/monitoring:test_resource 21:15:56 00:00 [main] (To run a reporting server: ./pants server) 21:15:56 00:00 [bootstrap] 21:15:57 00:01 [setup] 21:15:57 00:01 [parse] Executing tasks in goals: bootstrap - imports - gen - check-exclusives - resolve - compile - resources - test 21:15:57 00:01 [bootstrap] 21:15:57 00:01 [bootstrap-jvm-tools] 21:15:57 00:01 [imports] 21:15:57 00:01 [ivy-imports] 21:15:57 00:01 [gen] 21:15:57 00:01 [thrift] 21:15:57 00:01 [scrooge] 21:15:57 00:01 [protoc] 21:15:57 00:01 [antlr] 21:15:57 00:01 [ragel] 21:15:57 00:01 [jaxb] 21:15:57 00:01 [wire] 21:15:57 00:01 [aapt] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [resolve] 21:15:57 00:01 [ivy] 21:15:57 00:01 [compile] 21:15:57 00:01 [jvm] 21:15:57 00:01 [jvm-compilers] 21:15:57 00:01 [resources] 21:15:57 00:01 [prepare] 21:15:57 00:01 [test] 21:15:57 00:01 [run_prep_command] 21:15:57 00:01 [pytest] 21:15:57 00:01 [run] == test session starts === platform darwin -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4 plugins: cov, timeout collected 5 items src/test/python/apache/thermos/monitoring/test_resource.py . 5 passed in 0.18 seconds 21:16:04 00:08 [junit] 21:16:04 00:08 [specs] SUCCESS Thanks, Joe Smith
Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 4d4f3ab9442ceda912ac4ce07eb15e291a4ed08c src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 7, 2015, 2:27 a.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Fix checkstyle + legit flakes bug! Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - pants.ini 99648e46ca755c3c8e22d90c7b682107d8dee333 src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30647: Instrument the HealthChecker to export stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- (Updated Feb. 6, 2015, 11:13 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Changes --- Fix style violation. Bugs: AURORA-1062 https://issues.apache.org/jira/browse/AURORA-1062 Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs (updated) - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 src/test/python/apache/aurora/executor/test_thermos_executor.py c8fab307d17949a8157659c4b3944ec7520feb9d Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 27182: Add a test for the thermos resource module
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/#review71510 --- Out of curiosity, why this test now? I sort of see this code on its deathbed as soon Mesos will have disk enforcement built in. - Brian Wickman On Feb. 6, 2015, 7:59 p.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27182/ --- (Updated Feb. 6, 2015, 7:59 p.m.) Review request for Aurora, Kevin Sweeney and Brian Wickman. Repository: aurora Description --- Add a test for the thermos resource module Diffs - src/main/python/apache/thermos/monitoring/monitor.py 8f87f5ffc39c87e87ff78b941ea30df7138bd1ef src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_resource.py PRE-CREATION Diff: https://reviews.apache.org/r/27182/diff/ Testing --- [tw-mbp-jsmith incubator-aurora (yasumoto/thermos_test)]$ ./pants test.pytest ./src/test/python/apache/thermos/monitoring:test_resource 21:15:56 00:00 [main] (To run a reporting server: ./pants server) 21:15:56 00:00 [bootstrap] 21:15:57 00:01 [setup] 21:15:57 00:01 [parse] Executing tasks in goals: bootstrap - imports - gen - check-exclusives - resolve - compile - resources - test 21:15:57 00:01 [bootstrap] 21:15:57 00:01 [bootstrap-jvm-tools] 21:15:57 00:01 [imports] 21:15:57 00:01 [ivy-imports] 21:15:57 00:01 [gen] 21:15:57 00:01 [thrift] 21:15:57 00:01 [scrooge] 21:15:57 00:01 [protoc] 21:15:57 00:01 [antlr] 21:15:57 00:01 [ragel] 21:15:57 00:01 [jaxb] 21:15:57 00:01 [wire] 21:15:57 00:01 [aapt] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [check-exclusives] 21:15:57 00:01 [resolve] 21:15:57 00:01 [ivy] 21:15:57 00:01 [compile] 21:15:57 00:01 [jvm] 21:15:57 00:01 [jvm-compilers] 21:15:57 00:01 [resources] 21:15:57 00:01 [prepare] 21:15:57 00:01 [test] 21:15:57 00:01 [run_prep_command] 21:15:57 00:01 [pytest] 21:15:57 00:01 [run] == test session starts === platform darwin -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4 plugins: cov, timeout collected 5 items src/test/python/apache/thermos/monitoring/test_resource.py . 5 passed in 0.18 seconds 21:16:04 00:08 [junit] 21:16:04 00:08 [specs] SUCCESS Thanks, Joe Smith
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 6, 2015, 11:41 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Update updates[] to only map task_ids. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - examples/jobs/cron_hello_world.aurora a99b3ab7b1c82e4e8d72ac7dece6e44d1bba498f src/main/python/apache/aurora/executor/bin/BUILD 4d4f3ab9442ceda912ac4ce07eb15e291a4ed08c src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30741: Add an interface and implementations of PathDetector.
On Feb. 6, 2015, 9:44 p.m., Joshua Cohen wrote: src/main/python/apache/thermos/monitoring/detector.py, line 45 https://reviews.apache.org/r/30741/diff/2/?file=852426#file852426line45 Just for my own education, what's the difference between returning `self._paths` and `self._paths[:]`? self._paths[:] returns a copy of the list instead of the list itself. this prevents people from mutating the list and polluting subsequent calls. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/#review71505 --- On Feb. 6, 2015, 7:30 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/ --- (Updated Feb. 6, 2015, 7:30 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1024 https://issues.apache.org/jira/browse/AURORA-1024 Repository: aurora Description --- The goal here is that any place that uses a fixed checkpoint root that potentially needs to interact with multiple checkpoint roots (e.g. thermos observer, thermos cli, gc executor) will instead use a PathDetector implementation. The default implementation is just the FixedPathDetector to which you can pass --checkpoint_root. However in subsequent reviews, we will 1) plumb PathDetector into the abovementioned components and then 2) wire up a ChainedPathDetector that has both a FixedPathDetector and MesosPathDetector. Diffs - src/main/python/apache/aurora/executor/BUILD 79037bc5af67e5287afa2270b70992cac42de5a7 src/main/python/apache/aurora/executor/common/BUILD e64362e3b7d603f51b0545db414a3b3df414594f src/main/python/apache/aurora/executor/common/path_detector.py PRE-CREATION src/main/python/apache/aurora/executor/executor_detector.py 7b8fc4e1cf0b03f7d425a2f6bd0944b583c96737 src/main/python/apache/aurora/executor/gc_executor.py 952d77d8ff525ef069a921a987d056de81fc7476 src/main/python/apache/thermos/monitoring/BUILD 33259c854f7b28f772310620f9101fb304f30715 src/main/python/apache/thermos/monitoring/detector.py 117aef5b0f11dd33781b314be7c80cb4034cc9f8 src/test/python/apache/aurora/executor/BUILD aa3bc3b934ea40ee34241521e3f581b0d8ed0ab6 src/test/python/apache/aurora/executor/common/BUILD 9e3a657523fc9b848785fcbb0db2e737d371f08f src/test/python/apache/aurora/executor/common/test_path_detector.py PRE-CREATION src/test/python/apache/aurora/executor/test_executor_detector.py 85d541838c3f74afc022f4ea672e06b90e7e8142 src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/30741/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor/common:: src/test/python/apache/thermos/monitoring:: One failure due to incompatible merge w/ r/30704 -- will send follow up patch. Thanks, Brian Wickman
Re: Review Request 30741: Add an interface and implementations of PathDetector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/ --- (Updated Feb. 6, 2015, 10:40 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- address jcohen's feedback. Bugs: AURORA-1024 https://issues.apache.org/jira/browse/AURORA-1024 Repository: aurora Description --- The goal here is that any place that uses a fixed checkpoint root that potentially needs to interact with multiple checkpoint roots (e.g. thermos observer, thermos cli, gc executor) will instead use a PathDetector implementation. The default implementation is just the FixedPathDetector to which you can pass --checkpoint_root. However in subsequent reviews, we will 1) plumb PathDetector into the abovementioned components and then 2) wire up a ChainedPathDetector that has both a FixedPathDetector and MesosPathDetector. Diffs (updated) - src/main/python/apache/aurora/executor/BUILD 79037bc5af67e5287afa2270b70992cac42de5a7 src/main/python/apache/aurora/executor/common/BUILD e64362e3b7d603f51b0545db414a3b3df414594f src/main/python/apache/aurora/executor/common/path_detector.py PRE-CREATION src/main/python/apache/aurora/executor/executor_detector.py 7b8fc4e1cf0b03f7d425a2f6bd0944b583c96737 src/main/python/apache/aurora/executor/gc_executor.py 952d77d8ff525ef069a921a987d056de81fc7476 src/main/python/apache/thermos/monitoring/BUILD 33259c854f7b28f772310620f9101fb304f30715 src/main/python/apache/thermos/monitoring/detector.py 117aef5b0f11dd33781b314be7c80cb4034cc9f8 src/test/python/apache/aurora/executor/BUILD aa3bc3b934ea40ee34241521e3f581b0d8ed0ab6 src/test/python/apache/aurora/executor/common/BUILD 9e3a657523fc9b848785fcbb0db2e737d371f08f src/test/python/apache/aurora/executor/common/test_path_detector.py PRE-CREATION src/test/python/apache/aurora/executor/test_executor_detector.py 85d541838c3f74afc022f4ea672e06b90e7e8142 src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/30741/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor/common:: src/test/python/apache/thermos/monitoring:: One failure due to incompatible merge w/ r/30704 -- will send follow up patch. Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 7, 2015, 12:46 a.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Master merge. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30695: Implements log rotation in the Thermos runner.
On Feb. 6, 2015, 6:52 p.m., Brian Wickman wrote: This is super rad. Thanks for taking this on. Before I do a deeper dive, what do you think about making the logrotate policy be specified by the user instead of the framework owner, with a sensible default? For example, if this is configurable on the process object, you can have different policies per process, e.g. ```py class RotatePolicy(Struct): log_size = Default(Integer, 32*MB) backups = Default(Integer, 10) copytruncate = Default(Boolean, False) compress = Default(Boolean, False) hangup_command = String ... # union class Logger(Struct): standard = Boolean # standard i/o devnull = Boolean # /dev/null redirection logrotate = RotatePolicy # use a logrotation policy DefaultLogger = Logger(standard=True) class Process(Struct): cmdline = Required(String) name= Required(String) ... logger = Default(Logger, DefaultLogger) ``` This also means reduced end-to-end plumbing through all the binaries, class constructors, etc. And if you ever need to add new features (e.g. a compress option), they're fairly well encapsulated within the Logger union. George Sirois wrote: Awesome, thanks for the feedback. I'd be willing to take this on; it would definitely make the plumbing a lot cleaner and provide more flexibility, although the downside is that it's now harder to apply a universal default (besides whatever we arrive at as the Aurora default). I'll be able to pick this up next week and can probably have a modified review out by Wednesday evening. What do you think about starting out with a simple configuration (just log_size and backups on RotatePolicy) and iterating from there? I also have one question - what distinction are you making between the standard flag on Logger and the existence of a rotation policy? Yeah, all the extra parameters were just for illustration only. Not asking for any more functionality than what you already have since it already provides tremendous value. The idea for 'standard' in Logger is just to be explicit about current behavior (unrestricted logging to stdout/stderr) and use it as the default. As for applying a universal default that's not standard, there are a few ways that you could do this, from environment variables (THERMOS_FORCE_ROTATE? idk) to building an aurora client using a custom entry point that patches Process.TYPEMAP['logger'] to use a different default. Both are kind of sketch but within the realm of sketch found elsewhere in the code. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/#review71472 --- On Feb. 6, 2015, 5:51 p.m., George Sirois wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/ --- (Updated Feb. 6, 2015, 5:51 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-95 https://issues.apache.org/jira/browse/AURORA-95 Repository: aurora Description --- Implements log rotation in the Thermos runner. Diffs - docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/thermos_task_runner.py 9ff8c5379aad7ac05e44b1f5a2b74f759f26 src/main/python/apache/thermos/bin/thermos_runner.py bd8cf7f4cda54b6be72dad64f9446eedeb132211 src/main/python/apache/thermos/core/process.py 5ce138dab161d880c0bd58b87a6f5a54d4ca2f99 src/main/python/apache/thermos/core/runner.py f949f279a071c6464b026749f51afc776102f2aa src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 Diff: https://reviews.apache.org/r/30695/diff/ Testing --- ./pants test src/test/python/apache/thermos/core:all Thanks, George Sirois
Re: Review Request 30647: Instrument the HealthChecker to export stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- (Updated Feb. 6, 2015, 7:44 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Changes --- Remove no longer valid assertions about exported metrics. Bugs: AURORA-1062 https://issues.apache.org/jira/browse/AURORA-1062 Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs (updated) - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 src/test/python/apache/aurora/executor/test_thermos_executor.py c8fab307d17949a8157659c4b3944ec7520feb9d Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Review Request 30741: Add an interface and implementations of PathDetector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/ --- Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1024 https://issues.apache.org/jira/browse/AURORA-1024 Repository: aurora Description --- The goal here is that any place that uses a fixed checkpoint root that potentially needs to interact with multiple checkpoint roots (e.g. thermos observer, thermos cli, gc executor) will instead use a PathDetector implementation. The default implementation is just the FixedPathDetector to which you can pass --checkpoint_root. However in subsequent reviews, we will 1) plumb PathDetector into the abovementioned components and then 2) wire up a ChainedPathDetector that has both a FixedPathDetector and MesosPathDetector. Diffs - src/main/python/apache/aurora/executor/BUILD 79037bc5af67e5287afa2270b70992cac42de5a7 src/main/python/apache/aurora/executor/common/BUILD e64362e3b7d603f51b0545db414a3b3df414594f src/main/python/apache/aurora/executor/common/path_detector.py PRE-CREATION src/main/python/apache/aurora/executor/executor_detector.py 7b8fc4e1cf0b03f7d425a2f6bd0944b583c96737 src/main/python/apache/aurora/executor/gc_executor.py 952d77d8ff525ef069a921a987d056de81fc7476 src/main/python/apache/thermos/monitoring/BUILD 33259c854f7b28f772310620f9101fb304f30715 src/main/python/apache/thermos/monitoring/detector.py 117aef5b0f11dd33781b314be7c80cb4034cc9f8 src/test/python/apache/aurora/executor/BUILD aa3bc3b934ea40ee34241521e3f581b0d8ed0ab6 src/test/python/apache/aurora/executor/common/BUILD 9e3a657523fc9b848785fcbb0db2e737d371f08f src/test/python/apache/aurora/executor/common/test_path_detector.py PRE-CREATION src/test/python/apache/aurora/executor/test_executor_detector.py 85d541838c3f74afc022f4ea672e06b90e7e8142 src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/30741/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor/common:: src/test/python/apache/thermos/monitoring:: One failure due to incompatible merge w/ r/30704 -- will send follow up patch. Thanks, Brian Wickman
Re: Review Request 30695: Implements log rotation in the Thermos runner.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/#review71472 --- This is super rad. Thanks for taking this on. Before I do a deeper dive, what do you think about making the logrotate policy be specified by the user instead of the framework owner, with a sensible default? For example, if this is configurable on the process object, you can have different policies per process, e.g. ```py class RotatePolicy(Struct): log_size = Default(Integer, 32*MB) backups = Default(Integer, 10) copytruncate = Default(Boolean, False) compress = Default(Boolean, False) hangup_command = String ... # union class Logger(Struct): standard = Boolean # standard i/o devnull = Boolean # /dev/null redirection logrotate = RotatePolicy # use a logrotation policy DefaultLogger = Logger(standard=True) class Process(Struct): cmdline = Required(String) name= Required(String) ... logger = Default(Logger, DefaultLogger) ``` This also means reduced end-to-end plumbing through all the binaries, class constructors, etc. And if you ever need to add new features (e.g. a compress option), they're fairly well encapsulated within the Logger union. - Brian Wickman On Feb. 6, 2015, 5:51 p.m., George Sirois wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30695/ --- (Updated Feb. 6, 2015, 5:51 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-95 https://issues.apache.org/jira/browse/AURORA-95 Repository: aurora Description --- Implements log rotation in the Thermos runner. Diffs - docs/deploying-aurora-scheduler.md d1123359961fd59ddb8c1a07f80f293bdd46019f src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/thermos_task_runner.py 9ff8c5379aad7ac05e44b1f5a2b74f759f26 src/main/python/apache/thermos/bin/thermos_runner.py bd8cf7f4cda54b6be72dad64f9446eedeb132211 src/main/python/apache/thermos/core/process.py 5ce138dab161d880c0bd58b87a6f5a54d4ca2f99 src/main/python/apache/thermos/core/runner.py f949f279a071c6464b026749f51afc776102f2aa src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 Diff: https://reviews.apache.org/r/30695/diff/ Testing --- ./pants test src/test/python/apache/thermos/core:all Thanks, George Sirois
Re: Review Request 30741: Add an interface and implementations of PathDetector.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30741/ --- (Updated Feb. 6, 2015, 7:30 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Fix test failure. Bugs: AURORA-1024 https://issues.apache.org/jira/browse/AURORA-1024 Repository: aurora Description --- The goal here is that any place that uses a fixed checkpoint root that potentially needs to interact with multiple checkpoint roots (e.g. thermos observer, thermos cli, gc executor) will instead use a PathDetector implementation. The default implementation is just the FixedPathDetector to which you can pass --checkpoint_root. However in subsequent reviews, we will 1) plumb PathDetector into the abovementioned components and then 2) wire up a ChainedPathDetector that has both a FixedPathDetector and MesosPathDetector. Diffs (updated) - src/main/python/apache/aurora/executor/BUILD 79037bc5af67e5287afa2270b70992cac42de5a7 src/main/python/apache/aurora/executor/common/BUILD e64362e3b7d603f51b0545db414a3b3df414594f src/main/python/apache/aurora/executor/common/path_detector.py PRE-CREATION src/main/python/apache/aurora/executor/executor_detector.py 7b8fc4e1cf0b03f7d425a2f6bd0944b583c96737 src/main/python/apache/aurora/executor/gc_executor.py 952d77d8ff525ef069a921a987d056de81fc7476 src/main/python/apache/thermos/monitoring/BUILD 33259c854f7b28f772310620f9101fb304f30715 src/main/python/apache/thermos/monitoring/detector.py 117aef5b0f11dd33781b314be7c80cb4034cc9f8 src/test/python/apache/aurora/executor/BUILD aa3bc3b934ea40ee34241521e3f581b0d8ed0ab6 src/test/python/apache/aurora/executor/common/BUILD 9e3a657523fc9b848785fcbb0db2e737d371f08f src/test/python/apache/aurora/executor/common/test_path_detector.py PRE-CREATION src/test/python/apache/aurora/executor/test_executor_detector.py 85d541838c3f74afc022f4ea672e06b90e7e8142 src/test/python/apache/thermos/monitoring/BUILD 33d6bba43aff6d62b2646491f004475c27ed99db src/test/python/apache/thermos/monitoring/test_detector.py PRE-CREATION Diff: https://reviews.apache.org/r/30741/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor/common:: src/test/python/apache/thermos/monitoring:: One failure due to incompatible merge w/ r/30704 -- will send follow up patch. Thanks, Brian Wickman
Re: Review Request 30647: Instrument the HealthChecker to export stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/#review71484 --- whoops, there is a legit breakage due to the removal of the 'enabled' flags. will fix. - Brian Wickman On Feb. 5, 2015, 7:17 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- (Updated Feb. 5, 2015, 7:17 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Bugs: AURORA-1062 https://issues.apache.org/jira/browse/AURORA-1062 Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 30650: Upgrade virtualenv to 12.0.7
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30650/#review71453 --- Ship it! Ship It! - Brian Wickman On Feb. 5, 2015, 4:13 a.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30650/ --- (Updated Feb. 5, 2015, 4:13 a.m.) Review request for Aurora, Brian Wickman and Zameer Manji. Bugs: AURORA-979 https://issues.apache.org/jira/browse/AURORA-979 Repository: aurora Description --- Upgrade virtualenv to 12.0.7 virtualenv == [12.0.7 changelog](https://pypi.python.org/pypi/virtualenv/12.0.7) and selected highlights: * Upgrade pip to 6.0.8 * Upgrade setuptools to 12.0.5 * Revert several sys.path changes new in 12.0 which were breaking virtualenv. * **PROCESS** Version numbers are now simply ``X.Y`` where the leading ``1`` has been dropped. * Now using pytest framework * Correct sys.path ordering for debian, issue #461 * Correctly throws error on older Pythons, issue #619 * Allow for empty $PATH, pull #601 * Don't set prompt if $env:VIRTUAL_ENV_DISABLE_PROMPT is set for Powershell * Updated setuptools to 7.0 Diffs - build-support/virtualenv 0175f0e38e5f995ff4e8335fb86ec14869894e69 Diff: https://reviews.apache.org/r/30650/diff/ Testing --- ```sh $ ./pants test.pytest --no-fast ./src/test/python/apache/aurora:all ``` Thanks, Joe Smith
Re: Review Request 30647: Instrument the HealthChecker to export stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- (Updated Feb. 5, 2015, 7:15 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Changes --- Add coverage for snooze/latency. Bugs: AURORA-1062 https://issues.apache.org/jira/browse/AURORA-1062 Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs (updated) - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 30647: Instrument the HealthChecker to export stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- (Updated Feb. 5, 2015, 7:17 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Changes --- Actually compare latency metric. Bugs: AURORA-1062 https://issues.apache.org/jira/browse/AURORA-1062 Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs (updated) - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 30704: Remove default root in TaskPath.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30704/#review71352 --- Ship it! Ship It! - Brian Wickman On Feb. 6, 2015, 1:19 a.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30704/ --- (Updated Feb. 6, 2015, 1:19 a.m.) Review request for Aurora and Brian Wickman. Bugs: AURORA-1024 https://issues.apache.org/jira/browse/AURORA-1024 Repository: aurora Description --- To allow for an API that will allow for multiple thermos roots it would be best to remove the default root in TaskPath. This way code that does not explictly pass in a root will cause an exception. Diffs - src/main/python/apache/aurora/executor/bin/gc_executor_main.py 890bfaa28f1e72d283958659598befee8f47c7b7 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 9df9b4b79c0c7d29c5088409bf15c0d32a621df0 src/main/python/apache/aurora/executor/thermos_task_runner.py 7b346e253677ee9b42c57782f7f67ff63b6a0083 src/main/python/apache/thermos/bin/thermos.py ebf5488a76f8f1199972532118fda61b411df3bd src/main/python/apache/thermos/common/BUILD 6015f9e9a23f71bf6dede1f4698fe63dbb4dcfaa src/main/python/apache/thermos/common/constants.py PRE-CREATION src/main/python/apache/thermos/common/path.py 9e617051f16f4270b3958f48e0cc43706d245eec src/main/python/apache/thermos/observer/bin/thermos_observer.py 53c331c221b84bef4f2ab80829d51347c6232ad7 src/test/python/apache/thermos/common/test_pathspec.py 3437b196d33d7c2ff6ba292ff99b6881954e7ecb Diff: https://reviews.apache.org/r/30704/diff/ Testing --- ./pants test src/test/python/apache/thermos:: Thanks, Zameer Manji
Re: Review Request 30649: Upgrade pants to 0.0.28
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30649/#review71284 --- Ship it! Ship It! - Brian Wickman On Feb. 5, 2015, 4:13 a.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30649/ --- (Updated Feb. 5, 2015, 4:13 a.m.) Review request for Aurora, Brian Wickman and Zameer Manji. Bugs: AURORA-1104 https://issues.apache.org/jira/browse/AURORA-1104 Repository: aurora Description --- Upgrade pants to 0.0.28 pants = [0.0.28 changelog](https://pypi.python.org/pypi/pantsbuild.pants/0.0.28) and selected highlights: * bump virtualenv version to 12.0.5 RB #1621 * Make ‘setup-py’ show up in ‘./pants goal goals’ RB #1466 * Support use of pytest’s –pdb mode RB #1570 * Upgrade pex dependency to 0.8.4 Pick up several perf wins Pick up fix that allows pex to read older pexes RB #1648 RB #1693 * Refactor setting of PYTHONPATH in pants.ini RB #1586 Diffs - .pantsversion 24ff85581f81976c5f70fe1a8c3c0f62b5275c91 build-support/pants_requirements.txt fb9a2d2dd5f17b3523eea1e4a7a77d69feddff39 Diff: https://reviews.apache.org/r/30649/diff/ Testing --- Reviewed the diff [for twitter/commons](https://github.com/twitter/commons/commit/574d79891d74a3a142e1db6b085b4c7e13c67659#diff-570e3786705ee6965f4f2f72c715382bR194) ``` $ ./pants test.pytest --no-fast ./src/test/python/apache/aurora:all ``` Thanks, Joe Smith
Re: Review Request 30647: Instrument the HealthChecker to export stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- (Updated Feb. 5, 2015, 1:46 a.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Bugs: AURORA-1062 https://issues.apache.org/jira/browse/AURORA-1062 Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 30647: Instrument the HealthChecker to export stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- (Updated Feb. 5, 2015, 1:45 a.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Changes --- Rename health checker stats to health_checker. Add reviewers. Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs (updated) - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Review Request 30647: Instrument the HealthChecker to export stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30647/ --- Review request for Aurora. Repository: aurora Description --- Instrument the HealthChecker to export stats. HealthChecker plugin now should export three stats: consecutive_failures: number of consecutive failures experienced (resets on success) latency: how long health checks are taking in practice snoozed: whether or not the health checker is snoozed Diffs - src/main/python/apache/aurora/executor/common/health_checker.py 60676ba0fbd8a218fe4309f07de28e2c66d54530 src/main/python/apache/aurora/executor/common/status_checker.py 624921d68199df098ea51ee8a10815403bf58984 src/test/python/apache/aurora/executor/common/test_health_checker.py def249c2509a28f7145380f250f79202b653dc83 Diff: https://reviews.apache.org/r/30647/diff/ Testing --- ./pants test.pytest --no-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 30187: Remove support for cluster metadata in YAML format.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30187/#review70419 --- Ship it! Ship It! - Brian Wickman On Jan. 28, 2015, 8:26 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30187/ --- (Updated Jan. 28, 2015, 8:26 p.m.) Review request for Aurora, Brian Wickman and Zameer Manji. Bugs: AURORA-1029 https://issues.apache.org/jira/browse/AURORA-1029 Repository: aurora Description --- Remove support for cluster metadata in YAML format. Diffs - src/main/python/apache/aurora/common/clusters.py e55aa774b4b868f696a7de51bb016f950871dd1e src/test/python/apache/aurora/common/BUILD 14165b96be99b8de418f4bb8def9f27eaf29e67d src/test/python/apache/aurora/common/test_clusters.py 45250e609cca1149dc296b2aaf645ff2f58f8288 Diff: https://reviews.apache.org/r/30187/diff/ Testing --- ./build-support/jenkins/build.sh test_end_to_end.sh is currently broken on master, i will address that and ensure it passes before committing this. Thanks, Bill Farner
Re: Review Request 29286: Fix BUILD files in apache.aurora.admin and apache.aurora.client.api.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29286/#review70132 --- What specifically is taking place here? (I think I understand some of it, but I'm not sure what was broken or what is being improved.) - Brian Wickman On Dec. 20, 2014, 3:05 a.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29286/ --- (Updated Dec. 20, 2014, 3:05 a.m.) Review request for Aurora and Brian Wickman. Repository: aurora Description --- Fix BUILD files in apache.aurora.admin and apache.aurora.client.api. Diffs - src/main/python/apache/aurora/admin/BUILD f874264bdf07a9cbb2f0990739be3c95f851b040 src/main/python/apache/aurora/admin/host_maintenance.py 8fa5182fae509493def7175635fbe1cb79fa3eec src/main/python/apache/aurora/client/api/BUILD 65e5a85e23c4c698356c8b45c45943e560c1bcd5 src/main/python/apache/aurora/client/api/command_runner.py 48cb567c2098620e0ee322fe9528e167ce7c7c62 src/main/python/apache/aurora/client/api/disambiguator.py 6a78ccd44533ef327f751a08c9e2e16555354d97 src/test/python/apache/aurora/admin/BUILD 3a216809d1e31247f7d01451fcc7fd877a4c1fb2 src/test/python/apache/aurora/client/api/BUILD 2c0c4070cc1f1784b1d4e7f9cd8aac236e97be75 src/test/python/apache/aurora/client/api/test_task_util.py 048aff6874259810efea463df1ca2a1fdc419ca1 Diff: https://reviews.apache.org/r/29286/diff/ Testing --- ./pants src/test/python:all Thanks, Kevin Sweeney
Re: Review Request 30187: Remove support for cluster metadata in YAML format.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30187/#review69668 --- src/main/python/apache/aurora/common/clusters.py https://reviews.apache.org/r/30187/#comment114399 wait -- the # noqa is necessary, otherwise checkstyle will fail. - Brian Wickman On Jan. 22, 2015, 9:09 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30187/ --- (Updated Jan. 22, 2015, 9:09 p.m.) Review request for Aurora, Brian Wickman and Zameer Manji. Bugs: AURORA-1029 https://issues.apache.org/jira/browse/AURORA-1029 Repository: aurora Description --- Remove support for cluster metadata in YAML format. Diffs - src/main/python/apache/aurora/common/clusters.py e55aa774b4b868f696a7de51bb016f950871dd1e src/test/python/apache/aurora/common/BUILD 14165b96be99b8de418f4bb8def9f27eaf29e67d src/test/python/apache/aurora/common/test_clusters.py 45250e609cca1149dc296b2aaf645ff2f58f8288 Diff: https://reviews.apache.org/r/30187/diff/ Testing --- ./build-support/jenkins/build.sh test_end_to_end.sh is currently broken on master, i will address that and ensure it passes before committing this. Thanks, Bill Farner
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69480 --- this is looking great. just a few small nits and one higher level question re: where the schema should live. src/main/python/apache/aurora/executor/common/sandbox.py https://reviews.apache.org/r/28920/#comment114179 minor nit: snake_case, i.e. _create_symlinks src/main/python/apache/aurora/executor/common/sandbox.py https://reviews.apache.org/r/28920/#comment114182 os.chdir is a surprising side-effect for sandbox.create. could you instead put this in ThermosTaskRunner.start? i.e. where we do self._popen = subprocess.Popen(cmdline_args), Popen takes a cwd= keyword argument, and it seems like the presence of $MESOS_DIRECTORY should be enough signal that we should be setting it, and we don't need to chdir here. src/main/python/apache/aurora/executor/thermos_task_runner.py https://reviews.apache.org/r/28920/#comment114178 minor nit: os.path.join(X, 'sandbox') instead of + '/sandbox' src/main/python/apache/aurora/executor/thermos_task_runner.py https://reviews.apache.org/r/28920/#comment114181 b/c of AURORA-749, I think this can now just be: None if assigned_task.task.container.docker else assigned_task.task.job.role src/main/python/apache/thermos/config/schema_base.py https://reviews.apache.org/r/28920/#comment114180 sort of higher-level question -- since this information is not used by thermos at all, does it make sense in src/main/python/apache/aurora/config/schema/base.py instead? usually Aurora concerns are in Job() and Thermos concerns are in Task(). - Brian Wickman On Jan. 23, 2015, 5:23 p.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 23, 2015, 5:23 p.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh f692f025a7e5f2f0dddb7f6c81ea12fcb8272020 examples/vagrant/provision-dev-cluster.sh 40c165925c2110fb727c66ff5a34cf5ab8415343 examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 8428941c10641857a952f34df4e46a8fac5476a8 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py f29faf1d51baa4af66ad8c6579ffa354409e9536 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py
Re: Review Request 28920: Add support for docker containers to aurora
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/#review69505
---
Ship it!
sweet! ship it modulo minor nit below
src/main/python/apache/aurora/executor/thermos_task_runner.py
https://reviews.apache.org/r/28920/#comment114201
os.environ.get('MESOS_DIRECTORY')
(i worry that cwd='.' will actually break something, whereas cwd=None will
be a no-op.)
- Brian Wickman
On Jan. 24, 2015, midnight, Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/
---
(Updated Jan. 24, 2015, midnight)
Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner.
Bugs: AURORA-633
https://issues.apache.org/jira/browse/AURORA-633
Repository: aurora
Description
---
This change adds support for launching docker containers through aurora.
These changes are based off of the discussion in
https://issues.apache.org/jira/browse/AURORA-633
As of now, a special thermos_executor.sh script is needed to launch the
executor inside docker containers. A sample aurora file is in
examples/jobs/docker.
In addition, mesos-slave must be run with `--containerizers=docker,mesos`,
the example upstart config in examples/vagrant/upstart has been updated to
reflect this.
More information is in subsequent review request comments.
Diffs
-
Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc
api/src/main/thrift/org/apache/aurora/gen/api.thrift
08ba1cdf88b712de22c26c04443079282db59ef9
config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5
docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea
docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7
examples/jobs/docker/hello_docker.aurora PRE-CREATION
examples/vagrant/aurorabuild.sh f692f025a7e5f2f0dddb7f6c81ea12fcb8272020
examples/vagrant/provision-dev-cluster.sh
40c165925c2110fb727c66ff5a34cf5ab8415343
examples/vagrant/upstart/aurora-scheduler.conf
788ec254270bca074dae91829c7f4fccdc8f8bb0
examples/vagrant/upstart/mesos-slave.conf
512ce7ecf34042ed68dda55efb2dd0415f8469db
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java
8428941c10641857a952f34df4e46a8fac5476a8
src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java
5226e3d1b303b1773a057078f2911c5ec2aa97f5
src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java
d885b224ec5a1d529347d84e03ba98ab6734a126
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
01b03508afac37b5a8f0ec5c3da1460695e1ef59
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
5cc85f1f87f3b8355c89e8ecac19de1122a079e6
src/main/python/apache/aurora/config/schema/base.py
d7897794c736778983d506c337a1392f3cc0cc20
src/main/python/apache/aurora/config/thrift.py
ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c
src/main/python/apache/aurora/executor/aurora_executor.py
636b23d30a897b557eb8c3f8733c90b23cb807ef
src/main/python/apache/aurora/executor/bin/thermos_executor_main.py
f7d8977e42aa56188799400bf8e12a6886fb4a8f
src/main/python/apache/aurora/executor/common/sandbox.py
f47a32b3fefb4a89940b1ddc473b8316ac00df12
src/main/python/apache/aurora/executor/thermos_task_runner.py
f29faf1d51baa4af66ad8c6579ffa354409e9536
src/main/python/apache/thermos/core/runner.py
41200bd5d74ee3239279567d79a5f48fb6af2e44
src/main/resources/scheduler/assets/configSummary.html
28878908b0c9381e366b71a3135dfc28c542a890
src/main/resources/scheduler/assets/js/services.js
b744f375411e09b7f776e4a05ee5961227143439
src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java
9fd188fb8f004d0a7664420bfda56568cebedb6f
src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java
876e173ccbac04e4a06a245648c7c6af15eaaa92
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
c7de6e111300b009e1f9f430624a56100328184e
src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java
7eafe074b686d55ad96018006cf4acfa823513c3
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
ad9126c32893080e128d086ea3bfd7ad23d27b89
src/test/python/apache/aurora/client/cli/test_status.py
e531fa06e508d9792af51c62e67120c21baa7a81
src/test/python/apache/aurora/executor/test_thermos_executor.py
6c8ae1cf7fb578237708a954bd42162d66b39f4d
Diff: https://reviews.apache.org/r/28920/diff/
Testing
Re: Review Request 30187: Remove support for cluster metadata in YAML format.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30187/#review69269 --- I think we should leave the yaml code but drop the PyYAML dependency from the client requirements. (And inject it into the test -- possibly one test with and one test without to make sure the try/except also functions correctly.) I feel that .yml is actually a much more natural way of defining the cluster list than json, because YAML supports basic templating in order to reduce redundancy, similar to pystachio. The way the code is currently structured, YAML will still work fine if it's available in the environmenet of your Aurora client, and gracefully fall back if not. Thoughts? - Brian Wickman On Jan. 22, 2015, 9:09 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30187/ --- (Updated Jan. 22, 2015, 9:09 p.m.) Review request for Aurora, Brian Wickman and Zameer Manji. Bugs: AURORA-1029 https://issues.apache.org/jira/browse/AURORA-1029 Repository: aurora Description --- Remove support for cluster metadata in YAML format. Diffs - src/main/python/apache/aurora/common/clusters.py e55aa774b4b868f696a7de51bb016f950871dd1e src/test/python/apache/aurora/common/BUILD 14165b96be99b8de418f4bb8def9f27eaf29e67d src/test/python/apache/aurora/common/test_clusters.py 45250e609cca1149dc296b2aaf645ff2f58f8288 Diff: https://reviews.apache.org/r/30187/diff/ Testing --- ./build-support/jenkins/build.sh test_end_to_end.sh is currently broken on master, i will address that and ensure it passes before committing this. Thanks, Bill Farner
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69282 --- src/main/python/apache/thermos/core/runner.py https://reviews.apache.org/r/28920/#comment113921 this is an abstraction leak. grep the thermos codebase for 'aurora' and 'mesos'. thermos should remain aurora/mesos agnostic. thermos_runner.py includes a --sandbox flag and we should pass $MESOS_DIRECTORY to that, and not override here. the log_dir defaults to os.path.join(sandbox, '.logs'), so the log_dir part here is unnecessary. - Brian Wickman On Jan. 16, 2015, 12:08 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 16, 2015, 12:08 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 7eafe074b686d55ad96018006cf4acfa823513c3 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java ad9126c32893080e128d086ea3bfd7ad23d27b89 src/test/python/apache/aurora/client/cli/test_status.py e531fa06e508d9792af51c62e67120c21baa7a81 src/test/python/apache
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 22, 2015, 10:22 p.m., Brian Wickman wrote: src/main/python/apache/thermos/core/runner.py, lines 627-632 https://reviews.apache.org/r/28920/diff/18/?file=823218#file823218line627 this is an abstraction leak. grep the thermos codebase for 'aurora' and 'mesos'. thermos should remain aurora/mesos agnostic. thermos_runner.py includes a --sandbox flag and we should pass $MESOS_DIRECTORY to that, and not override here. the log_dir defaults to os.path.join(sandbox, '.logs'), so the log_dir part here is unnecessary. Steve Niemitz wrote: Heh, I'm pretty sure it was like that ~20 reviews ago. To be clear, your proposing passing the flag into the runner from thermos_task_runner.py? yep. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69282 --- On Jan. 16, 2015, 12:08 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 16, 2015, 12:08 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java
Re: Review Request 28920: Add support for docker containers to aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/#review69288 --- src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java https://reviews.apache.org/r/28920/#comment113931 can't the stuff in DOCKER_COMMAND_PREFIX be accomplished with a DockerSandboxProvider in the thermos executor? similarly, DOCKER_COMMAND_SUFFIX should be unnecessary since this information is available in the TaskConfig. if both the above are true, then we don't need any of the changes to CommandUtil. - Brian Wickman On Jan. 16, 2015, 12:08 a.m., Steve Niemitz wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28920/ --- (Updated Jan. 16, 2015, 12:08 a.m.) Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner. Bugs: AURORA-633 https://issues.apache.org/jira/browse/AURORA-633 Repository: aurora Description --- This change adds support for launching docker containers through aurora. These changes are based off of the discussion in https://issues.apache.org/jira/browse/AURORA-633 As of now, a special thermos_executor.sh script is needed to launch the executor inside docker containers. A sample aurora file is in examples/jobs/docker. In addition, mesos-slave must be run with `--containerizers=docker,mesos`, the example upstart config in examples/vagrant/upstart has been updated to reflect this. More information is in subsequent review request comments. Diffs - Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc api/src/main/thrift/org/apache/aurora/gen/api.thrift 08ba1cdf88b712de22c26c04443079282db59ef9 config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5 docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7 examples/jobs/docker/hello_docker.aurora PRE-CREATION examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd examples/vagrant/provision-dev-cluster.sh 7af4b52a6876268a97630279221bb98d9b04efad examples/vagrant/upstart/aurora-scheduler.conf 788ec254270bca074dae91829c7f4fccdc8f8bb0 examples/vagrant/upstart/mesos-slave.conf 512ce7ecf34042ed68dda55efb2dd0415f8469db src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 5226e3d1b303b1773a057078f2911c5ec2aa97f5 src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java d885b224ec5a1d529347d84e03ba98ab6734a126 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 01b03508afac37b5a8f0ec5c3da1460695e1ef59 src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5bf283062c9d119ff91ed45da8b236e36d0fc9aa src/main/python/apache/aurora/config/thrift.py ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c src/main/python/apache/aurora/executor/aurora_executor.py 636b23d30a897b557eb8c3f8733c90b23cb807ef src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f7d8977e42aa56188799400bf8e12a6886fb4a8f src/main/python/apache/aurora/executor/common/sandbox.py f47a32b3fefb4a89940b1ddc473b8316ac00df12 src/main/python/apache/aurora/executor/thermos_task_runner.py 5e4bd65537d186459003c0b9434f1b769e04f448 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/runner.py 8aac6b50c66080abbb5308b367e9f74c487f42e3 src/main/resources/scheduler/assets/configSummary.html 28878908b0c9381e366b71a3135dfc28c542a890 src/main/resources/scheduler/assets/js/services.js b744f375411e09b7f776e4a05ee5961227143439 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java 876e173ccbac04e4a06a245648c7c6af15eaaa92 src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java dc2cb37adf32df0a6e4c7ee2ba776ba9f1f3c2f8 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ddcb511d108220ab5e4efcf3496458f7ab4a20c2 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 7eafe074b686d55ad96018006cf4acfa823513c3 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java ad9126c32893080e128d086ea3bfd7ad23d27b89 src/test/python/apache/aurora/client/cli/test_status.py e531fa06e508d9792af51c62e67120c21baa7a81 src/test/python/apache
Re: Review Request 28920: Add support for docker containers to aurora
On Jan. 22, 2015, 10:42 p.m., Brian Wickman wrote:
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java,
lines 153-158
https://reviews.apache.org/r/28920/diff/18/?file=823211#file823211line153
can't the stuff in DOCKER_COMMAND_PREFIX be accomplished with a
DockerSandboxProvider in the thermos executor?
similarly, DOCKER_COMMAND_SUFFIX should be unnecessary since this
information is available in the TaskConfig.
if both the above are true, then we don't need any of the changes to
CommandUtil.
Steve Niemitz wrote:
No unfortunatly, once the executor is started it's too late to change
some things, they need to be setup before the executor launches (see my
previous comment w/ Kevin). The suffix could be moved into the executor, but
again I'd like to keep docker specific logic out of there.
it's definitely not too late. and i disagree that we should keep docker
specific logic out of the executor. that's *exactly* where it should be.
afaict, the only thing affected is LogOptions.set_log_dir('.') in
thermos_executor_main. this can easily be changed to
LogOptions.set_log_dir(os.getenv('MESOS_DIRECTORY') or '.')
*ideally* in my opinion, the logical conclusion is creating a
thermos_docker_executor_main that wires things up in accordance to the docker
philosophy, then have a completely separate docker task factory within the
scheduler to construct these. but that might be a little more unsavory in the
short term than just having the thermos_executor that understands both
containerized and non-containerized modes. there is precedence for this
already (exactly why we have sandbox providers and task providers in the first
place -- to switch implementations entirely based upon incoming TaskConfigs.)
- Brian
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/#review69288
---
On Jan. 16, 2015, 12:08 a.m., Steve Niemitz wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28920/
---
(Updated Jan. 16, 2015, 12:08 a.m.)
Review request for Aurora, Jay Buffington, Kevin Sweeney, and Bill Farner.
Bugs: AURORA-633
https://issues.apache.org/jira/browse/AURORA-633
Repository: aurora
Description
---
This change adds support for launching docker containers through aurora.
These changes are based off of the discussion in
https://issues.apache.org/jira/browse/AURORA-633
As of now, a special thermos_executor.sh script is needed to launch the
executor inside docker containers. A sample aurora file is in
examples/jobs/docker.
In addition, mesos-slave must be run with `--containerizers=docker,mesos`,
the example upstart config in examples/vagrant/upstart has been updated to
reflect this.
More information is in subsequent review request comments.
Diffs
-
Vagrantfile f8b7db8eebdc6a10989de3bc9a2c3e89ce17f5fc
api/src/main/thrift/org/apache/aurora/gen/api.thrift
08ba1cdf88b712de22c26c04443079282db59ef9
config/legacy_untested_classes.txt 33c1d6eb4ea02e01b7002c2c2bae5a3858c8b0e5
docs/configuration-reference.md f3cb257206a194b82fd2045dc20456ee832dbcea
docs/deploying-aurora-scheduler.md 711ae7eda07c2c1735601c265c06a88c1862cce7
examples/jobs/docker/hello_docker.aurora PRE-CREATION
examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd
examples/vagrant/provision-dev-cluster.sh
7af4b52a6876268a97630279221bb98d9b04efad
examples/vagrant/upstart/aurora-scheduler.conf
788ec254270bca074dae91829c7f4fccdc8f8bb0
examples/vagrant/upstart/mesos-slave.conf
512ce7ecf34042ed68dda55efb2dd0415f8469db
src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java
72c7545e7f16549f6a9ccb5fb74a06f154a7ea94
src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java
5226e3d1b303b1773a057078f2911c5ec2aa97f5
src/main/java/org/apache/aurora/scheduler/base/CommandUtil.java
d885b224ec5a1d529347d84e03ba98ab6734a126
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
01b03508afac37b5a8f0ec5c3da1460695e1ef59
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
5bf283062c9d119ff91ed45da8b236e36d0fc9aa
src/main/python/apache/aurora/config/thrift.py
ba94ac3c0cbaf3c91eb1a1d86a244ed6fa3b649c
src/main/python/apache/aurora/executor/aurora_executor.py
636b23d30a897b557eb8c3f8733c90b23cb807ef
src/main/python/apache/aurora/executor/bin/thermos_executor_main.py
f7d8977e42aa56188799400bf8e12a6886fb4a8f
src/main/python/apache/aurora/executor/common/sandbox.py
f47a32b3fefb4a89940b1ddc473b8316ac00df12
src/main/python
Re: Review Request 29984: Minor clean up to mock usage in resource manager integration tests.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29984/#review68961 --- Ship it! Ship It! - Brian Wickman On Jan. 16, 2015, 9:45 p.m., Joshua Cohen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29984/ --- (Updated Jan. 16, 2015, 9:45 p.m.) Review request for Aurora, Kevin Sweeney and Brian Wickman. Repository: aurora Description --- Minor clean up to mock usage in resource manager integration tests. We didn't need to wrap the mock disk collector in a lambda, just rather than asserting on the mock class, we instead needed to assert on its return value (the instance). Diffs - src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py bb3045700c2870329e175db44873dcb990226c7a Diff: https://reviews.apache.org/r/29984/diff/ Testing --- ./pants build src/test/python/apache/aurora/executor/common:resource_manager_integration Thanks, Joshua Cohen
Re: Review Request 29901: De-flakify resource_manager_integration test
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29901/ --- (Updated Jan. 16, 2015, 7:10 p.m.) Review request for Aurora, Joshua Cohen and Kevin Sweeney. Changes --- Assert on call_count. Repository: aurora Description --- De-flakify ResourceManager integration test. Diffs (updated) - src/main/python/apache/aurora/executor/common/resource_manager.py bf69e7e60ae4dd857243d8b040a70daca008ea28 src/main/python/apache/thermos/monitoring/resource.py 7b5bbe01810008c4d39b9dfee6521af01021a7ca src/test/python/apache/aurora/executor/common/BUILD 395ced27998def18eace2cbd46c736404c3d6411 src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py e3c766f9b7bf989c0790281231957947d2cda52a Diff: https://reviews.apache.org/r/29901/diff/ Testing --- ./pants goal test --no-test-pytest-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 29866: Upgrade pants to 0.0.27
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29866/ --- (Updated Jan. 16, 2015, 7:16 p.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Changes --- Remove changes to ResourceManager test. Repository: aurora Description --- Ports aurora repo to use pants 0.0.27 using the new pants new command processor. Diffs (updated) - .pantsversion 43b29618309bc6aa441848513d5dd3554a32c7c1 build-support/jenkins/build.sh 8de19cd865f3ae36c7ddc1aa0a8b453f95541b50 build-support/pants_requirements.txt 9a790f973cc7560980d43d5a1097338ff88a0397 build-support/python/make-pycharm-virtualenv 85e8137e8cab9765bd78ada6dedc9c60cc88e8a0 build-support/python/update-pants-requirements 523a3cb7e87146dc393d9ab09b23f2e3fb683b27 build-support/release/make-python-sdists 2f437d33928cc429a79aedb00247d472953fbb95 docs/developing-aurora-client.md a7253d2beba61f7aca94bab944f544b969f507fd examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd pants 40ddcf3a2c7359686ad326dd3d8ee37df55790e6 src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 src/test/python/apache/aurora/executor/test_thermos_task_runner.py f8cf9fc7f0ed58be2bd2af5124e1a5df0dd2f00c Diff: https://reviews.apache.org/r/29866/diff/ Testing --- bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Brian Wickman
Re: Review Request 29901: De-flakify resource_manager_integration test
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29901/ --- (Updated Jan. 15, 2015, 11:36 p.m.) Review request for Aurora and Kevin Sweeney. Changes --- Make flaky test less bad (it's still bad -- but this code is destined for the trash can once we have feature parity from mesos.) Repository: aurora Description --- De-flakify ResourceManager integration test. Diffs (updated) - src/main/python/apache/aurora/executor/common/resource_manager.py bf69e7e60ae4dd857243d8b040a70daca008ea28 src/main/python/apache/thermos/monitoring/resource.py 7b5bbe01810008c4d39b9dfee6521af01021a7ca src/test/python/apache/aurora/executor/common/BUILD 395ced27998def18eace2cbd46c736404c3d6411 src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py e3c766f9b7bf989c0790281231957947d2cda52a Diff: https://reviews.apache.org/r/29901/diff/ Testing --- ./pants goal test --no-test-pytest-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 29901: De-flakify resource_manager_integration test
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29901/ --- (Updated Jan. 16, 2015, 12:11 a.m.) Review request for Aurora and Kevin Sweeney. Changes --- Switch to using autospec, also allowing us to remove the factory stuff. Repository: aurora Description --- De-flakify ResourceManager integration test. Diffs (updated) - src/main/python/apache/aurora/executor/common/resource_manager.py bf69e7e60ae4dd857243d8b040a70daca008ea28 src/main/python/apache/thermos/monitoring/resource.py 7b5bbe01810008c4d39b9dfee6521af01021a7ca src/test/python/apache/aurora/executor/common/BUILD 395ced27998def18eace2cbd46c736404c3d6411 src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py e3c766f9b7bf989c0790281231957947d2cda52a Diff: https://reviews.apache.org/r/29901/diff/ Testing --- ./pants goal test --no-test-pytest-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 29901: De-flakify resource_manager_integration test
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29901/ --- (Updated Jan. 16, 2015, 12:51 a.m.) Review request for Aurora, Joshua Cohen and Kevin Sweeney. Changes --- Address Kevin's feedback. Repository: aurora Description --- De-flakify ResourceManager integration test. Diffs (updated) - src/main/python/apache/aurora/executor/common/resource_manager.py bf69e7e60ae4dd857243d8b040a70daca008ea28 src/main/python/apache/thermos/monitoring/resource.py 7b5bbe01810008c4d39b9dfee6521af01021a7ca src/test/python/apache/aurora/executor/common/BUILD 395ced27998def18eace2cbd46c736404c3d6411 src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py e3c766f9b7bf989c0790281231957947d2cda52a Diff: https://reviews.apache.org/r/29901/diff/ Testing --- ./pants goal test --no-test-pytest-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
Re: Review Request 29866: Upgrade pants to 0.0.27
On Jan. 14, 2015, 12:49 a.m., Kevin Sweeney wrote: src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py, line 21 https://reviews.apache.org/r/29866/diff/3/?file=820203#file820203line21 same, this doesn't appear to be related to a pants upgrade Brian Wickman wrote: this is to fix the test failure brought up by reviewbot. unclear if it's related to the pants upgrade, but it should be fixed regardless. Brian Wickman wrote: would you prefer that this go into a separate review? Kevin Sweeney wrote: Yes, I prefer to keep commits small and focused (if I revert a commit titled upgrade pants I don't expect it to also break an unrelated test). Sorry if my comment made that unclear. Joshua Cohen wrote: If the failure is caused by the pants upgrade, then it should ship w/ this review. If it's just an oversight from the initial commit of the resource manager, then split it out? Well, reverting this change won't break that test -- the test passes on master and breaks with the pants upgrade. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29866/#review67988 --- On Jan. 14, 2015, 12:15 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29866/ --- (Updated Jan. 14, 2015, 12:15 a.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Repository: aurora Description --- Ports aurora repo to use pants 0.0.27 using the new pants new command processor. Diffs - .pantsversion 43b29618309bc6aa441848513d5dd3554a32c7c1 build-support/jenkins/build.sh 8de19cd865f3ae36c7ddc1aa0a8b453f95541b50 build-support/pants_requirements.txt 9a790f973cc7560980d43d5a1097338ff88a0397 build-support/python/make-pycharm-virtualenv 85e8137e8cab9765bd78ada6dedc9c60cc88e8a0 build-support/python/update-pants-requirements 523a3cb7e87146dc393d9ab09b23f2e3fb683b27 build-support/release/make-python-sdists 2f437d33928cc429a79aedb00247d472953fbb95 docs/developing-aurora-client.md a7253d2beba61f7aca94bab944f544b969f507fd examples/vagrant/aurorabuild.sh 1e31f21998d02fd69ce0db88e6adb3d32cff67fd pants 40ddcf3a2c7359686ad326dd3d8ee37df55790e6 src/main/python/apache/aurora/executor/common/resource_manager.py bf69e7e60ae4dd857243d8b040a70daca008ea28 src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py e3c766f9b7bf989c0790281231957947d2cda52a src/test/python/apache/aurora/executor/test_thermos_executor.py 503e62f4cac872b14f6985b5bccc3e4dfcf81789 src/test/python/apache/aurora/executor/test_thermos_task_runner.py f8cf9fc7f0ed58be2bd2af5124e1a5df0dd2f00c Diff: https://reviews.apache.org/r/29866/diff/ Testing --- bash src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Brian Wickman
Review Request 29901: De-flakify resource_manager_integration test
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29901/ --- Review request for Aurora and Kevin Sweeney. Repository: aurora Description --- De-flakify ResourceManager integration test. Diffs - src/main/python/apache/aurora/executor/common/resource_manager.py bf69e7e60ae4dd857243d8b040a70daca008ea28 src/test/python/apache/aurora/executor/common/test_resource_manager_integration.py e3c766f9b7bf989c0790281231957947d2cda52a Diff: https://reviews.apache.org/r/29901/diff/ Testing --- ./pants goal test --no-test-pytest-fast src/test/python/apache/aurora/executor/common:: Thanks, Brian Wickman
