Re: Review Request 32013: Cache TaskDetectors for reuse.
On March 17, 2015, 7:49 p.m., Zameer Manji wrote: src/main/python/apache/aurora/executor/common/path_detector.py, line 34 https://reviews.apache.org/r/32013/diff/3/?file=896451#file896451line34 What's the rationale behind this change? I think the callers should be doing the checking for existance and dealing with missing directories as needed. The rationale is that some tasks will have checkpointing in the sandbox (0.9.0+) and some will have it outside the sandbox (0.8.0, or 0.9.0+ but whose thermos runners haven't yet been forked.) A typical production slave has 1000-2000 leftover framework directories, but since we haven't rolled sandbox checkpointing yet, that's 1000-2000 TaskDetectors we must construct and query if we don't filter by tasks that have this feature enabled. That filtering definitely seems like it belongs in the MesosPathDetector. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/#review76784 --- On March 17, 2015, 8:24 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 17, 2015, 8:24 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- SSIA Diffs - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c src/test/python/apache/aurora/executor/common/test_path_detector.py 2d6edae420c1c5b51fe93f97e04c24976afacd51 Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Re: Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/#review76800 --- Ship it! Master (4b39d26) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On March 17, 2015, 8:24 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 17, 2015, 8:24 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- SSIA Diffs - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c src/test/python/apache/aurora/executor/common/test_path_detector.py 2d6edae420c1c5b51fe93f97e04c24976afacd51 Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Re: Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 17, 2015, 8:24 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- rebase against master Repository: aurora Description --- SSIA Diffs (updated) - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c src/test/python/apache/aurora/executor/common/test_path_detector.py 2d6edae420c1c5b51fe93f97e04c24976afacd51 Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Re: Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/#review76794 --- Ship it! Ship It! - Joe Smith On March 17, 2015, 1:24 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 17, 2015, 1:24 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- SSIA Diffs - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c src/test/python/apache/aurora/executor/common/test_path_detector.py 2d6edae420c1c5b51fe93f97e04c24976afacd51 Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Re: Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/#review76667 --- This patch does not apply cleanly on master (1b2a343), do you need to rebase? I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On March 16, 2015, 10:37 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 16, 2015, 10:37 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- SSIA Diffs - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c src/test/python/apache/aurora/executor/common/test_path_detector.py 2d6edae420c1c5b51fe93f97e04c24976afacd51 src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Re: Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 16, 2015, 9:47 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- Patch os.path.exists in the executor path detector test. Repository: aurora Description --- SSIA Diffs (updated) - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/common/path.py 4359a5143551fff3cf66e39e68ba9ba02b8d6e2e src/main/python/apache/thermos/monitoring/detector.py e0922e61ef03fc578988f992aba46570ea1b5d6c src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c src/test/python/apache/aurora/executor/common/test_path_detector.py 2d6edae420c1c5b51fe93f97e04c24976afacd51 src/test/python/apache/thermos/monitoring/BUILD 4e99da403f322940579fd4f9412e022df0790884 src/test/python/apache/thermos/monitoring/test_detector.py e9005c402ec81bbb415d9c814aa7b4e759138304 Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Re: Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/#review76658 --- Master (1b2a343) is red with this patch. ./build-support/jenkins/build.sh src.test.python.apache.aurora.client.hooks.non_hooked_api . SUCCESS src.test.python.apache.aurora.common.test_aurora_job_key . SUCCESS src.test.python.apache.aurora.common.test_cluster . SUCCESS src.test.python.apache.aurora.common.test_cluster_option . SUCCESS src.test.python.apache.aurora.common.test_clusters . SUCCESS src.test.python.apache.aurora.common.test_http_signaler . SUCCESS src.test.python.apache.aurora.common.test_pex_version . SUCCESS src.test.python.apache.aurora.common.test_shellify . SUCCESS src.test.python.apache.aurora.common.test_transport . SUCCESS src.test.python.apache.aurora.config.test_base . SUCCESS src.test.python.apache.aurora.config.test_constraint_parsing . SUCCESS src.test.python.apache.aurora.config.test_loader . SUCCESS src.test.python.apache.aurora.config.test_thrift . SUCCESS src.test.python.apache.aurora.executor.common.announcer . SUCCESS src.test.python.apache.aurora.executor.common.directory_sandbox . SUCCESS src.test.python.apache.aurora.executor.common.executor_detector . SUCCESS src.test.python.apache.aurora.executor.common.executor_timeout . SUCCESS src.test.python.apache.aurora.executor.common.health_checker . SUCCESS src.test.python.apache.aurora.executor.common.kill_manager . SUCCESS src.test.python.apache.aurora.executor.common.path_detector . SUCCESS src.test.python.apache.aurora.executor.common.status_checker . SUCCESS src.test.python.apache.aurora.executor.common.task_info . SUCCESS src.test.python.apache.aurora.executor.executor_base . SUCCESS src.test.python.apache.aurora.executor.executor_vars . SUCCESS src.test.python.apache.aurora.executor.gc_executor . SUCCESS src.test.python.apache.aurora.executor.status_manager . SUCCESS src.test.python.apache.aurora.executor.thermos_executor . SUCCESS src.test.python.apache.aurora.executor.thermos_task_runner . SUCCESS src.test.python.apache.thermos.cli.commands.commands . SUCCESS src.test.python.apache.thermos.cli.common . SUCCESS src.test.python.apache.thermos.cli.main . SUCCESS src.test.python.apache.thermos.common.test_pathspec . SUCCESS src.test.python.apache.thermos.core.test_process . FAILURE src.test.python.apache.thermos.core.test_runner_integration . SUCCESS src.test.python.apache.thermos.monitoring.test_disk . SUCCESS FAILURE [31m FAILURE[0m I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On March 16, 2015, 9:47 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 16, 2015, 9:47 p.m.) Review request for Aurora, Joe Smith and Zameer Manji.
Re: Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 16, 2015, 10:37 p.m.) Review request for Aurora, Joe Smith and Zameer Manji. Changes --- Incorrectly parented the last review off master instead of 32012 Repository: aurora Description --- SSIA Diffs (updated) - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c src/test/python/apache/aurora/executor/common/test_path_detector.py 2d6edae420c1c5b51fe93f97e04c24976afacd51 src/test/python/apache/thermos/core/test_process.py e261249b977802851ffc3d89437761c532fcd3f8 Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- SSIA Diffs - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
Re: Review Request 32013: Cache TaskDetectors for reuse.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32013/#review76318
---
Master (ee1a13a) is red with this patch.
./build-support/jenkins/build.sh
plugins: cov, timeout
[1m
collecting 0 items[0m[1m
collecting 1 items[0m[1m
collected 1 items
[0m
src/test/python/apache/aurora/executor/common/test_path_detector.py::test_path_detector
[31mFAILED[0m
FAILURES
___ test_path_detector ___
[1mdef test_path_detector():[0m
[1m ROOTS = ('/var/lib/mesos1/slaves',
'/var/lib/mesos2/slaves')[0m
[1m FAKE_ROOT = '/var/blah/blah'[0m
[1m FAKE_CHECKPOINT_DIR = 'ckpt'[0m
[1m[0m
[1m path1, path2 = ([0m
[1m ExecutorDetector.path(Match(ROOTS[0],
'slave001', 'framework1', 'executor1', 'latest')),[0m
[1m ExecutorDetector.path(Match(ROOTS[1],
'slave002', 'framework2', 'executor2', 'latest')),[0m
[1m )[0m
[1m[0m
[1m with mock.patch('glob.glob',
return_value=(path1, path2)) as glob:[0m
[1mmpd = MesosPathDetector(root=FAKE_ROOT,
sandbox_path=FAKE_CHECKPOINT_DIR)[0m
[1mpaths = list(mpd.get_paths())[0m
[1m assert len(paths) == 2[0m
[1m[31mE assert 0 == 2[0m
[1m[31mE+ where 0 = len([])[0m
src/test/python/apache/aurora/executor/common/test_path_detector.py:45:
AssertionError
generated xml file:
/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/test-results/src.test.python.apache.aurora.executor.common.path_detector.xml
[1m[31m 1 failed in 0.06 seconds
[0m
src.test.python.apache.aurora.client.hooks.non_hooked_api
. SUCCESS
src.test.python.apache.aurora.executor.common.path_detector
. FAILURE
src.test.python.apache.thermos.cli.commands.commands
. SUCCESS
src.test.python.apache.thermos.cli.common
. SUCCESS
src.test.python.apache.thermos.cli.main
. SUCCESS
src.test.python.apache.thermos.common.test_pathspec
. SUCCESS
src.test.python.apache.thermos.monitoring.test_disk
. SUCCESS
FAILURE
[31m
FAILURE[0m
I will refresh this build result if you post a review containing @ReviewBot
retry
- Aurora ReviewBot
On March 13, 2015, 12:48 a.m., Brian Wickman wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32013/
---
(Updated March 13, 2015, 12:48 a.m.)
Review request for Aurora, Joe Smith and Zameer Manji.
Repository: aurora
Description
---
SSIA
Diffs
-
src/main/python/apache/aurora/executor/common/path_detector.py
e4135cca09b43e204c52774b747f74212d0a5732
src/main/python/apache/thermos/observer/detector.py
5347cded7061b085b135d62fdb3a5c4a75f81e5c
Diff: https://reviews.apache.org/r/32013/diff/
Testing
---
./pants test src/test/python/apache/thermos/observer:: + manual observation
Thanks,
Brian Wickman
Re: Review Request 32013: Cache TaskDetectors for reuse.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/#review76319 --- whoops, forgot that I added os.path.exists filtering -- will update tests. - Brian Wickman On March 13, 2015, 12:48 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32013/ --- (Updated March 13, 2015, 12:48 a.m.) Review request for Aurora, Joe Smith and Zameer Manji. Repository: aurora Description --- SSIA Diffs - src/main/python/apache/aurora/executor/common/path_detector.py e4135cca09b43e204c52774b747f74212d0a5732 src/main/python/apache/thermos/observer/detector.py 5347cded7061b085b135d62fdb3a5c4a75f81e5c Diff: https://reviews.apache.org/r/32013/diff/ Testing --- ./pants test src/test/python/apache/thermos/observer:: + manual observation Thanks, Brian Wickman
