Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review73337 --- @Reviewbot retry - Brian Wickman On Feb. 20, 2015, 7:06 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 20, 2015, 7:06 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 6530f4914736754f92ba192c1a345e4b7e4a5398 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py dbec82ffe4e155059f3e6b1aa0e67ec4c52a9611 src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py 27dee7fa10a4141ec7e9f4440bde2dd257db1cc6 Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
On Feb. 20, 2015, 11:11 p.m., Brian Wickman wrote: @Reviewbot retry Is that test known to be flaky? If so, can you file a ticket? - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review73337 --- On Feb. 20, 2015, 7:06 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 20, 2015, 7:06 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 6530f4914736754f92ba192c1a345e4b7e4a5398 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py dbec82ffe4e155059f3e6b1aa0e67ec4c52a9611 src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py 27dee7fa10a4141ec7e9f4440bde2dd257db1cc6 Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review73383 --- Ship it! Master (e5de618) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 20, 2015, 7:06 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 20, 2015, 7:06 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 6530f4914736754f92ba192c1a345e4b7e4a5398 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py dbec82ffe4e155059f3e6b1aa0e67ec4c52a9611 src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py 27dee7fa10a4141ec7e9f4440bde2dd257db1cc6 Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
On Feb. 20, 2015, 11:11 p.m., Brian Wickman wrote: @Reviewbot retry Bill Farner wrote: Is that test known to be flaky? If so, can you file a ticket? Brian Wickman wrote: https://reviews.apache.org/r/30647/ should deflake it. Gotcha. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review73337 --- On Feb. 20, 2015, 7:06 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 20, 2015, 7:06 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 6530f4914736754f92ba192c1a345e4b7e4a5398 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py dbec82ffe4e155059f3e6b1aa0e67ec4c52a9611 src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py 27dee7fa10a4141ec7e9f4440bde2dd257db1cc6 Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
On Feb. 10, 2015, 7:12 p.m., Joshua Cohen wrote: src/main/python/apache/thermos/monitoring/garbage.py, line 120 https://reviews.apache.org/r/30749/diff/9/?file=859049#file859049line120 Are we no longer worried about the scenario where the checkpoint root is $HOME or /? The checkpoint root could be set to this but only maliciously. On Feb. 10, 2015, 7:12 p.m., Joshua Cohen wrote: src/test/python/apache/aurora/executor/test_gc_executor.py, line 180 https://reviews.apache.org/r/30749/diff/9/?file=859053#file859053line180 s/'fake_root'/FAKE_ROOT fixed On Feb. 10, 2015, 7:12 p.m., Joshua Cohen wrote: src/test/python/apache/aurora/executor/test_gc_executor.py, lines 210-211 https://reviews.apache.org/r/30749/diff/9/?file=859053#file859053line210 Am I missing something or is this a duplicate of the make_task defined above? totally a dupe, deleted. On Feb. 10, 2015, 7:12 p.m., Joshua Cohen wrote: src/test/python/apache/aurora/executor/test_gc_executor.py, line 254 https://reviews.apache.org/r/30749/diff/9/?file=859053#file859053line254 nit: maybe give task_id a default value in make_task instead of having to always pass in the same placeholder value? Nah, because in all of these assertions we actually need to know the task id so it wouldn't make sense to omit from make_task(). - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71833 --- On Feb. 20, 2015, 7:06 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 20, 2015, 7:06 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 6530f4914736754f92ba192c1a345e4b7e4a5398 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py dbec82ffe4e155059f3e6b1aa0e67ec4c52a9611 src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py 27dee7fa10a4141ec7e9f4440bde2dd257db1cc6 Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
On Feb. 10, 2015, 7:19 p.m., Joe Smith wrote: src/main/python/apache/aurora/executor/gc_executor.py, line 73 https://reviews.apache.org/r/30749/diff/9/?file=859047#file859047line73 Some documentation would go a long way here. I assume this is the way to go from `task_id` - `path_of_the_checkpoint_stream_on_disk` ? Added some commentary. On Feb. 10, 2015, 7:19 p.m., Joe Smith wrote: src/main/python/apache/aurora/executor/gc_executor.py, line 129 https://reviews.apache.org/r/30749/diff/9/?file=859047#file859047line129 s/PathDetector/path_detector, I think whoops, thanks. On Feb. 10, 2015, 7:19 p.m., Joe Smith wrote: src/main/python/apache/aurora/executor/gc_executor.py, line 135 https://reviews.apache.org/r/30749/diff/9/?file=859047#file859047line135 ``` :param task: an instance of a task to retrieve checkpoint path :type task: RootedTask instance ``` Does it make sense to pydoc a private method? i'll add it but i also don't want to necessarily set a precedent. On Feb. 10, 2015, 7:19 p.m., Joe Smith wrote: src/main/python/apache/thermos/monitoring/garbage.py, line 31 https://reviews.apache.org/r/30749/diff/9/?file=859049#file859049line31 Mind adding a docstring here and for the `__init__` ? done On Feb. 10, 2015, 7:19 p.m., Joe Smith wrote: src/main/python/apache/thermos/monitoring/garbage.py, line 173 https://reviews.apache.org/r/30749/diff/9/?file=859049#file859049line173 Everytime I see a namedtuple I cringe a bit (since I really only see them as a poor replacement for spec'd mocks) but if you don't mind adding docs here that'd be ~alright. This code is on the chopping block, so I wouldn't lament the general practices here too much. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71831 --- On Feb. 9, 2015, 11:16 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 11:16 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review73303 --- Master (e0e3f2e) is red with this patch. ./build-support/jenkins/build.sh src.test.python.apache.aurora.client.cli.sla . SUCCESS src.test.python.apache.aurora.client.cli.supdate . SUCCESS src.test.python.apache.aurora.client.cli.task . SUCCESS src.test.python.apache.aurora.client.cli.update . SUCCESS src.test.python.apache.aurora.client.cli.version . SUCCESS src.test.python.apache.aurora.client.config . SUCCESS src.test.python.apache.aurora.client.factory . SUCCESS src.test.python.apache.aurora.client.hooks.hooked_api . SUCCESS src.test.python.apache.aurora.client.hooks.non_hooked_api . SUCCESS src.test.python.apache.aurora.common.test_aurora_job_key . SUCCESS src.test.python.apache.aurora.common.test_cluster . SUCCESS src.test.python.apache.aurora.common.test_cluster_option . SUCCESS src.test.python.apache.aurora.common.test_clusters . SUCCESS src.test.python.apache.aurora.common.test_http_signaler . SUCCESS src.test.python.apache.aurora.common.test_pex_version . SUCCESS src.test.python.apache.aurora.common.test_shellify . SUCCESS src.test.python.apache.aurora.common.test_transport . SUCCESS src.test.python.apache.aurora.config.test_base . SUCCESS src.test.python.apache.aurora.config.test_constraint_parsing . SUCCESS src.test.python.apache.aurora.config.test_loader . SUCCESS src.test.python.apache.aurora.config.test_thrift . SUCCESS src.test.python.apache.aurora.executor.common.announcer . SUCCESS src.test.python.apache.aurora.executor.common.directory_sandbox . SUCCESS src.test.python.apache.aurora.executor.common.executor_detector . SUCCESS src.test.python.apache.aurora.executor.common.executor_timeout . SUCCESS src.test.python.apache.aurora.executor.common.health_checker . FAILURE src.test.python.apache.aurora.executor.common.path_detector . SUCCESS src.test.python.apache.aurora.executor.common.task_info . SUCCESS src.test.python.apache.aurora.executor.executor_base . SUCCESS src.test.python.apache.aurora.executor.executor_vars . SUCCESS src.test.python.apache.aurora.executor.status_manager . SUCCESS src.test.python.apache.aurora.executor.thermos_task_runner . SUCCESS src.test.python.apache.thermos.common.test_pathspec . SUCCESS src.test.python.apache.thermos.core.test_runner_integration . SUCCESS src.test.python.apache.thermos.monitoring.test_disk . SUCCESS FAILURE [31m FAILURE[0m I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 20, 2015, 7:06 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 20, 2015, 7:06 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji.
Re: Review Request 30749: Port GC executor to PathDetector interface
On Feb. 20, 2015, 11:11 p.m., Brian Wickman wrote: @Reviewbot retry Bill Farner wrote: Is that test known to be flaky? If so, can you file a ticket? https://reviews.apache.org/r/30647/ should deflake it. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review73337 --- On Feb. 20, 2015, 7:06 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 20, 2015, 7:06 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 6530f4914736754f92ba192c1a345e4b7e4a5398 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py dbec82ffe4e155059f3e6b1aa0e67ec4c52a9611 src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py 27dee7fa10a4141ec7e9f4440bde2dd257db1cc6 Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71831 --- src/main/python/apache/aurora/executor/bin/BUILD https://reviews.apache.org/r/30749/#comment117696 +100 src/main/python/apache/aurora/executor/gc_executor.py https://reviews.apache.org/r/30749/#comment117690 Some documentation would go a long way here. I assume this is the way to go from `task_id` - `path_of_the_checkpoint_stream_on_disk` ? src/main/python/apache/aurora/executor/gc_executor.py https://reviews.apache.org/r/30749/#comment117691 s/PathDetector/path_detector, I think src/main/python/apache/aurora/executor/gc_executor.py https://reviews.apache.org/r/30749/#comment117693 ``` :param task: an instance of a task to retrieve checkpoint path :type task: RootedTask instance ``` src/main/python/apache/thermos/monitoring/garbage.py https://reviews.apache.org/r/30749/#comment117697 Mind adding a docstring here and for the `__init__` ? src/main/python/apache/thermos/monitoring/garbage.py https://reviews.apache.org/r/30749/#comment117698 Thanks for this cleanup! src/main/python/apache/thermos/monitoring/garbage.py https://reviews.apache.org/r/30749/#comment117700 Everytime I see a namedtuple I cringe a bit (since I really only see them as a poor replacement for spec'd mocks) but if you don't mind adding docs here that'd be ~alright. - Joe Smith On Feb. 9, 2015, 3:16 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 3:16 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71741 --- Ship it! Ship It! - Zameer Manji On Feb. 9, 2015, 3:16 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 3:16 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 6:45 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Revert unintended pants.ini change. Pull in a slice of r/30768 to pass e2e. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/aurora/executor/thermos_task_runner.py 9ff8c5379aad7ac05e44b1f5a2b74f759f26 src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71678 --- Ship it! There is a lot of churn because of interface changes but this change LGTM. - Zameer Manji On Feb. 9, 2015, 10:56 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 10:56 a.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 6:56 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Post master merge. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
On Feb. 8, 2015, 2:20 a.m., Zameer Manji wrote: src/main/python/apache/aurora/executor/gc_executor.py, line 73 https://reviews.apache.org/r/30749/diff/4/?file=853457#file853457line73 What's the benefit of using a namedtuple here? As opposed to? This means we don't have to change every function signature in this file. (I suppose on the other hand we get less type safety because functions called w/o the correct number of parameters would at least TypeError.) On Feb. 8, 2015, 2:20 a.m., Zameer Manji wrote: src/main/python/apache/aurora/executor/gc_executor.py, line 411 https://reviews.apache.org/r/30749/diff/4/?file=853457#file853457line411 Shouldn't this log messsage be inside the collector implementation? The gc executor is on its deathbed -- I propose not changing anything not completely necessary for its deprecation. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71567 --- On Feb. 7, 2015, 2:27 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 7, 2015, 2:27 a.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - pants.ini 99648e46ca755c3c8e22d90c7b682107d8dee333 src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71671 --- This patch does not apply cleanly on master (dcae1e8), do you need to rebase? I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 9, 2015, 6:45 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 6:45 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/aurora/executor/thermos_task_runner.py 9ff8c5379aad7ac05e44b1f5a2b74f759f26 src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 8:24 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Argh, can't catch a break. Fixes build typo. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 8:22 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Adds entry point tests for gc/thermos executors, since I got some of the import wiring wrong: vagrant@192:/var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/latest$ cat stderr WARNING: Logging before InitGoogleLogging() is written to STDERR I0209 19:16:58.301468 22286 fetcher.cpp:76] Fetching URI '/home/vagrant/aurora/dist/gc_executor.pex' I0209 19:16:58.301730 22286 fetcher.cpp:179] Copying resource from '/home/vagrant/aurora/dist/gc_executor.pex' to '/var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/6f1f2749-66a9-41e7-bb68-5e31c08cac6d' Traceback (most recent call last): File /var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/6f1f2749-66a9-41e7-bb68-5e31c08cac6d/gc_executor.pex/.bootstrap/_pex/pex.py, line 272, in execute self.execute_entry(entry_point, args) File /var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/6f1f2749-66a9-41e7-bb68-5e31c08cac6d/gc_executor.pex/.bootstrap/_pex/pex.py, line 320, in execute_entry runner(entry_point) File /var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/6f1f2749-66a9-41e7-bb68-5e31c08cac6d/gc_executor.pex/.bootstrap/_pex/pex.py, line 342, in execute_pkg_resources runner = entry.load(require=False) # trust that the environment is sane File /var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/6f1f2749-66a9-41e7-bb68-5e31c08cac6d/gc_executor.pex/.bootstrap/pkg_resources.py, line 2048, in load entry = __import__(self.module_name, globals(),globals(), ['__name__']) File /var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/6f1f2749-66a9-41e7-bb68-5e31c08cac6d/gc_executor.pex/apache/aurora/executor/bin/gc_executor_main.py, line 27, in module ImportError: No module named executor_detector vagrant@192:/var/lib/mesos/slaves/20150209-182237-119646400-5050-15827-S0/frameworks/20150209-182237-119646400-5050-15827-/executors/aurora.gc/runs/latest$ Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30749/#review71705
---
Master (dcae1e8) is red with this patch.
./build-support/jenkins/build.sh
Running setup.py install for twitter.common.collections
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.collections-0.3.0-py2.7-nspkg.pth
Running setup.py install for twitter.common.util
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.util-0.3.0-py2.7-nspkg.pth
Running setup.py install for twitter.common.log
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.log-0.3.0-py2.7-nspkg.pth
Running setup.py install for twitter.common.process
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.process-0.3.0-py2.7-nspkg.pth
Running setup.py install for gitdb
building 'gitdb._perf' extension
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2
-Wall -Wstrict-prototypes -fPIC -Igitdb -I/usr/include/python2.7 -c
gitdb/_fun.c -o build/temp.linux-x86_64-2.7/gitdb/_fun.o
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2
-Wall -Wstrict-prototypes -fPIC -Igitdb -I/usr/include/python2.7 -c
gitdb/_delta_apply.c -o build/temp.linux-x86_64-2.7/gitdb/_delta_apply.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
-Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv
-O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector
--param=ssp-buffer-size=4 -Wformat -Werror=format-security
build/temp.linux-x86_64-2.7/gitdb/_fun.o
build/temp.linux-x86_64-2.7/gitdb/_delta_apply.o -o
build/lib.linux-x86_64-2.7/gitdb/_perf.so
Running setup.py install for twitter.common.app
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.app-0.3.0-py2.7-nspkg.pth
Running setup.py install for GitPython
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/local/lib/python2.7/site-packages/setuptools/dist.py:292:
UserWarning: The version specified ('0.3.2 RC1') is an invalid version, this
may not work as expected with newer versions of setuptools, pip, and PyPI.
Please see PEP 440 for more details.
details. % self.metadata.version
Running setup.py install for pep8
Installing pep8 script to
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/bin
Running setup.py install for pyflakes
Installing pyflakes script to
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/bin
Running setup.py
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 11:16 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Checkstyle Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71720 --- Ship it! Master (68aa285) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 9, 2015, 11:16 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 11:16 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 0752d50015b2ff936f079c4a9f2777172dc00a93 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/bin/BUILD PRE-CREATION src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py PRE-CREATION src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71676 --- Ship it! Master (dcae1e8) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 9, 2015, 6:56 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 9, 2015, 6:56 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71567 --- pants.ini https://reviews.apache.org/r/30749/#comment117327 This not does not appear to be a change related to your work here. src/main/python/apache/aurora/executor/gc_executor.py https://reviews.apache.org/r/30749/#comment117328 What's the benefit of using a namedtuple here? src/main/python/apache/aurora/executor/gc_executor.py https://reviews.apache.org/r/30749/#comment117329 Shouldn't this log messsage be inside the collector implementation? - Zameer Manji On Feb. 6, 2015, 6:27 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 6, 2015, 6:27 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - pants.ini 99648e46ca755c3c8e22d90c7b682107d8dee333 src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - src/main/python/apache/aurora/executor/bin/BUILD 4d4f3ab9442ceda912ac4ce07eb15e291a4ed08c src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30749/#review71538
---
Master (11a65d2) is red with this patch.
./build-support/jenkins/build.sh
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.util-0.3.0-py2.7-nspkg.pth
Running setup.py install for twitter.common.log
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.log-0.3.0-py2.7-nspkg.pth
Running setup.py install for twitter.common.process
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.process-0.3.0-py2.7-nspkg.pth
Running setup.py install for gitdb
building 'gitdb._perf' extension
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2
-Wall -Wstrict-prototypes -fPIC -Igitdb -I/usr/include/python2.7 -c
gitdb/_fun.c -o build/temp.linux-x86_64-2.7/gitdb/_fun.o
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2
-Wall -Wstrict-prototypes -fPIC -Igitdb -I/usr/include/python2.7 -c
gitdb/_delta_apply.c -o build/temp.linux-x86_64-2.7/gitdb/_delta_apply.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
-Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv
-O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector
--param=ssp-buffer-size=4 -Wformat -Werror=format-security
build/temp.linux-x86_64-2.7/gitdb/_fun.o
build/temp.linux-x86_64-2.7/gitdb/_delta_apply.o -o
build/lib.linux-x86_64-2.7/gitdb/_perf.so
Running setup.py install for twitter.common.app
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/common/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.common.app-0.3.0-py2.7-nspkg.pth
Running setup.py install for GitPython
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/local/lib/python2.7/site-packages/setuptools/dist.py:292:
UserWarning: The version specified ('0.3.2 RC1') is an invalid version, this
may not work as expected with newer versions of setuptools, pip, and PyPI.
Please see PEP 440 for more details.
details. % self.metadata.version
Running setup.py install for pep8
Installing pep8 script to
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/bin
Running setup.py install for pyflakes
Installing pyflakes script to
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/bin
Running setup.py install for twitter.checkstyle
Skipping installation of
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter/__init__.py
(namespace package)
Installing
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/lib/python2.7/site-packages/twitter.checkstyle-0.1.0-py2.7-nspkg.pth
Installing twitterstyle script to
/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/python/checkstyle.venv/bin
Successfully installed GitPython-0.3.2rc1 gitdb-0.6.4 pep8-1.4.5 pyflakes-0.7.2
smmap-0.9.0 twitter.checkstyle-0.1.0 twitter.common.app-0.3.0
twitter.common.collections-0.3.0 twitter.common.contextutil-0.3.0
twitter.common.dirutil-0.3.0 twitter.common.lang-0.3.0 twitter.common.log-0.3.0
twitter.common.options-0.3.0
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 7, 2015, 2:27 a.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Fix checkstyle + legit flakes bug! Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - pants.ini 99648e46ca755c3c8e22d90c7b682107d8dee333 src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 6, 2015, 11:41 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Update updates[] to only map task_ids. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - examples/jobs/cron_hello_world.aurora a99b3ab7b1c82e4e8d72ac7dece6e44d1bba498f src/main/python/apache/aurora/executor/bin/BUILD 4d4f3ab9442ceda912ac4ce07eb15e291a4ed08c src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/#review71544 --- Ship it! Master (11a65d2) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 7, 2015, 2:27 a.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 7, 2015, 2:27 a.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs - pants.ini 99648e46ca755c3c8e22d90c7b682107d8dee333 src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
Re: Review Request 30749: Port GC executor to PathDetector interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30749/ --- (Updated Feb. 7, 2015, 12:46 a.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Changes --- Master merge. Bugs: AURORA-1025 https://issues.apache.org/jira/browse/AURORA-1025 Repository: aurora Description --- This makes the GC executor detect checkpoint roots via the PathDetector interface. This paves the way to detecting checkpoint roots both from /var/run/thermos and from /var/lib/mesos/** Diffs (updated) - src/main/python/apache/aurora/executor/bin/BUILD 1fa1e9e839bf28093b4e9ded403a761cf4bf5f44 src/main/python/apache/aurora/executor/bin/gc_executor_main.py b903bcb3630a8a8d50a2008bfae532b2eb947356 src/main/python/apache/aurora/executor/gc_executor.py 43b415bc6c5177be24ec036cc32ae7cbd87fc70f src/main/python/apache/thermos/bin/thermos.py 161bbdbc4de95c82e2b596e77b0eac7b920eae66 src/main/python/apache/thermos/monitoring/garbage.py 69bf8e4c2e2e5f85f6b822fbe45f828d61814d7f src/test/python/apache/aurora/executor/test_gc_executor.py b1bbc89a822302d8ea12324eb767631326639ebb Diff: https://reviews.apache.org/r/30749/diff/ Testing --- ./pants test.pytest --no-fast src/main/python:: Thanks, Brian Wickman
