Re: Review Request 37049: Integrating DelayExecutor into the scheduler's transaction handling.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37049/#review93983 --- Ship it! Master (7fdcfef) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Aug. 3, 2015, 11:49 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37049/ --- (Updated Aug. 3, 2015, 11:49 p.m.) Review request for Aurora and Kevin Sweeney. Bugs: AURORA-1395 https://issues.apache.org/jira/browse/AURORA-1395 Repository: aurora Description --- This creates two fundamental ways of performing asynchronous work in the scheduler: - Consume `@AsyncExecutor DelayExecutor` to perform work in the context of a database transaction, completing the work after the transaction - Consume `@BackgroundWorker ScheduledExecutorService` to perform work entirely in the background Diffs - src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java c345c926512431e447649689d152b1ffa876ee85 src/main/java/org/apache/aurora/scheduler/async/GatedDelayExecutor.java 1893a9bcbf6167177d8d0e83ec8ea4c10da899f0 src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 4b8a55fe5530c0b3e483838035cd1ee0f00a2c71 src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java fa9a09c19ad56cb0c13460464c6445cfe326c89b src/main/java/org/apache/aurora/scheduler/reconciliation/KillRetry.java 1611a3b69776ef72ed29fa2bd73df26442183a26 src/main/java/org/apache/aurora/scheduler/reconciliation/ReconciliationModule.java 406c07767b2bd4f4d80b9f0af72ae1cb4e457bc0 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 653e52b340edd6c87c12b0fdf760edf6d9d62722 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskTimeout.java fb8397260406cd45fcdfec17872b6ad4a011f2c5 src/main/java/org/apache/aurora/scheduler/scheduling/TaskGroups.java e60daad8606a7060f690f9a75d9c5bce0107522c src/main/java/org/apache/aurora/scheduler/scheduling/TaskThrottler.java e54e6c4b5b43b604110da11680b002f6d4b12df9 src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java ed92661605e27673c176b91535e4773be2a920a3 src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java a1f0d3ca89189a7791e823fec708b41a164cb7e0 src/main/java/org/apache/aurora/scheduler/storage/db/DbUtil.java 3a86614767cf0435add237901b12d8108be09970 src/test/java/org/apache/aurora/scheduler/async/AsyncModuleTest.java 5384307a6e4bf27800ab9e5104d0e74ac748e515 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java 91b91bc7bbaeaa5341565a8ee1ac3a97ea8a1620 src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java e725e102c4321c3aa30d89581783744475897f94 src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 088a4a6e874acc4926414bc30806e45fb961adec src/test/java/org/apache/aurora/scheduler/pruning/TaskHistoryPrunerTest.java 532b0ea70504b8b13f5e0ada5a519635c804a4b7 src/test/java/org/apache/aurora/scheduler/reconciliation/KillRetryTest.java 26f65fa99eab0a4d56d3e2da16e5704223c53780 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskTimeoutTest.java 97d25f9b20e49783e1576910883201f6e37f584f src/test/java/org/apache/aurora/scheduler/scheduling/TaskGroupsTest.java 55aad3572e3b93a73085dc94b135b63b81454cbc src/test/java/org/apache/aurora/scheduler/scheduling/TaskThrottlerTest.java 405502122325927aa90b36a89c55667042e01ca7 src/test/java/org/apache/aurora/scheduler/storage/db/DbStorageTest.java a4bcdd7ad06da21e17c43a53854658049e03eb0e src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java e198c1caca201d2a599404341d3ada6fae8f1b9d Diff: https://reviews.apache.org/r/37049/diff/ Testing --- Thanks, Bill Farner
Review Request 37049: Integrating DelayExecutor into the scheduler's transaction handling.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37049/ --- Review request for Aurora and Kevin Sweeney. Bugs: AURORA-1395 https://issues.apache.org/jira/browse/AURORA-1395 Repository: aurora Description --- This creates two fundamental ways of performing asynchronous work in the scheduler: - Consume `@AsyncExecutor DelayExecutor` to perform work in the context of a database transaction, completing the work after the transaction - Consume `@BackgroundWorker ScheduledExecutorService` to perform work entirely in the background Diffs - src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java c345c926512431e447649689d152b1ffa876ee85 src/main/java/org/apache/aurora/scheduler/async/GatedDelayExecutor.java 1893a9bcbf6167177d8d0e83ec8ea4c10da899f0 src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 4b8a55fe5530c0b3e483838035cd1ee0f00a2c71 src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java fa9a09c19ad56cb0c13460464c6445cfe326c89b src/main/java/org/apache/aurora/scheduler/reconciliation/KillRetry.java 1611a3b69776ef72ed29fa2bd73df26442183a26 src/main/java/org/apache/aurora/scheduler/reconciliation/ReconciliationModule.java 406c07767b2bd4f4d80b9f0af72ae1cb4e457bc0 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 653e52b340edd6c87c12b0fdf760edf6d9d62722 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskTimeout.java fb8397260406cd45fcdfec17872b6ad4a011f2c5 src/main/java/org/apache/aurora/scheduler/scheduling/TaskGroups.java e60daad8606a7060f690f9a75d9c5bce0107522c src/main/java/org/apache/aurora/scheduler/scheduling/TaskThrottler.java e54e6c4b5b43b604110da11680b002f6d4b12df9 src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java ed92661605e27673c176b91535e4773be2a920a3 src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java a1f0d3ca89189a7791e823fec708b41a164cb7e0 src/main/java/org/apache/aurora/scheduler/storage/db/DbUtil.java 3a86614767cf0435add237901b12d8108be09970 src/test/java/org/apache/aurora/scheduler/async/AsyncModuleTest.java 5384307a6e4bf27800ab9e5104d0e74ac748e515 src/test/java/org/apache/aurora/scheduler/http/JettyServerModuleTest.java 91b91bc7bbaeaa5341565a8ee1ac3a97ea8a1620 src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java e725e102c4321c3aa30d89581783744475897f94 src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 088a4a6e874acc4926414bc30806e45fb961adec src/test/java/org/apache/aurora/scheduler/pruning/TaskHistoryPrunerTest.java 532b0ea70504b8b13f5e0ada5a519635c804a4b7 src/test/java/org/apache/aurora/scheduler/reconciliation/KillRetryTest.java 26f65fa99eab0a4d56d3e2da16e5704223c53780 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskTimeoutTest.java 97d25f9b20e49783e1576910883201f6e37f584f src/test/java/org/apache/aurora/scheduler/scheduling/TaskGroupsTest.java 55aad3572e3b93a73085dc94b135b63b81454cbc src/test/java/org/apache/aurora/scheduler/scheduling/TaskThrottlerTest.java 405502122325927aa90b36a89c55667042e01ca7 src/test/java/org/apache/aurora/scheduler/storage/db/DbStorageTest.java a4bcdd7ad06da21e17c43a53854658049e03eb0e src/test/java/org/apache/aurora/scheduler/testing/FakeScheduledExecutor.java e198c1caca201d2a599404341d3ada6fae8f1b9d Diff: https://reviews.apache.org/r/37049/diff/ Testing --- Thanks, Bill Farner
Review Request 37041: Remove defunct scheduler-local.sh.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37041/ --- Review request for Aurora and Maxim Khutornenko. Repository: aurora Description --- Remove defunct scheduler-local.sh. Diffs - examples/scheduler/scheduler-local.sh 5b5d2fc3db6e24e38e9f73eff5f14224b81fed4f Diff: https://reviews.apache.org/r/37041/diff/ Testing --- Confirmed unused: ``` $ grep -RI 'scheduler-local.sh' . | grep -v '.git' | wc -l 0 ``` Thanks, Bill Farner
Re: Review Request 36972: One python_library per exported setup.py project
On July 31, 2015, 3:21 p.m., Brian Wickman wrote: I think it makes sense to split into two binary-exporting packages: one client-side and one server-side. The client side (aurora.client) should contain aurora client and aurora admin client. The server-side (reuse aurora.executor?) should contain aurora executor, thermos cli, thermos observer and thermos runner. Kevin Sweeney wrote: I like this suggestion and would like to refactor the code to fit it into this framework. Would you accept that in a follow-up review? Specifically there would then be 3 top-level packages (5 if you count the thrift API bindings, which I would also like to export to allow folks to write clients) ``` src/main/python/apache/aurora common/ client/ executor/ ``` ``` % pip install apache.aurora.executor # thermos # thermos-executor # thermos-observer # thermos-runner % pip install apache.aurora.client # aurora # aurora-admin ``` Everything else would remain more-or-less organized the same but move to a subpackage of one of the top-level targets. The rule is simple - if both server-side and client-side import something it goes under common/, otherwise it goes under client/ or executor/ as appropriate. Brian Wickman wrote: This seems reasonable. I'd prefer to keep the name aurora-executor but it probably merits public debate. The name aurora-executor is fine with me, in this case I've left it unchanged as thermos_executor. But I agree a debate on the proposed renaming can take place on another review. - Kevin --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36972/#review93793 --- On Aug. 1, 2015, 3:38 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36972/ --- (Updated Aug. 1, 2015, 3:38 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-1381 https://issues.apache.org/jira/browse/AURORA-1381 Repository: aurora Description --- This is a mostly red code diff and makes the pants build work similar to how it already works in an IDE environment while minimizing duplicated information. This change also removes the confusing versions of `thermos` and `thermos_observer` in favor of new ones. Because of the way covering dependencies were required it was easier to do this refactor than more tactically create a working `setup_py` target for `apache.aurora.tools` without cycles and without duplicated files. * Remove the `apache.thermos` package. * Rename the `apache.gen.aurora` package to `apache.aurora.thrift`. * Rename the `apache.gen.thermos` package to `apache.thermos.thrift`. * Introduce a new `apache.aurora.tools` package with the `thermos` and `thermos_observer` binaries. * Create apache.thermos.runner package, rename `thermos/bin` to `thermos/runner`. * Remove all `*-packaged` and virtual dependency trees, as well as the phrase covering dependencies. * Use `_`-prefix naming convention for private target names. * Replace manual list of targets in `make-python-sdists` with automatically-generated one (using new convention). * Introduce a new `apache.aurora.kerberos` package with the `kaurora` and `kaurora_admin` binaries. * Remove all `BUILD` files in `src/main` that don't contain an exported `setup.py` library (except one mentioned in TODO). * Use dictionary syntax in `with_binaries`. * Remove unused/unreachable binaries (most `bin/` dirs under `thermos/`). * Create blank `__init__.py` files in directories that pants warns aren't packages. * Added `thermos` command to vagrant image. Diffs - NEWS 0a8500c524a92eaf51384125687128d2fbbb0b53 api/src/main/thrift/org/apache/aurora/gen/BUILD d196fefc2c5e5ee32d0cf9c901cffe7d247379d1 api/src/main/thrift/org/apache/thermos/BUILD d0d789a6ee3971e3070f9397d53929563a77f7ea build-support/packaging/debian/rules 17e00c02a3cb3294d5107516d795a73587ca4f70 build-support/packaging/rpm/aurora.spec 7cf8de6f7b99788ca461a90fc4aefba7dccd7b63 build-support/python/make-pycharm-virtualenv 05a16d0d421982bcfb2e649be7b83a17d813efb1 build-support/release/make-python-sdists 9608f68e16243da01434ce2fc7d61bb7c7efd712 docs/README.md 9893763cb75faf1aeb66ed905e9f696d4b532d16 docs/build-system.md PRE-CREATION examples/vagrant/aurorabuild.sh fbaa6ae9ef7ff2910af8c9c0d6b8ef90ea3e152a src/main/python/apache/aurora/admin/BUILD 22bf3f9943ea11258ba681bdb80feb00206bb926 src/main/python/apache/aurora/admin/aurora_admin.py
Re: Review Request 36972: One python_library per exported setup.py project
On July 31, 2015, 10:21 p.m., Brian Wickman wrote: I think it makes sense to split into two binary-exporting packages: one client-side and one server-side. The client side (aurora.client) should contain aurora client and aurora admin client. The server-side (reuse aurora.executor?) should contain aurora executor, thermos cli, thermos observer and thermos runner. Kevin Sweeney wrote: I like this suggestion and would like to refactor the code to fit it into this framework. Would you accept that in a follow-up review? Specifically there would then be 3 top-level packages (5 if you count the thrift API bindings, which I would also like to export to allow folks to write clients) ``` src/main/python/apache/aurora common/ client/ executor/ ``` ``` % pip install apache.aurora.executor # thermos # thermos-executor # thermos-observer # thermos-runner % pip install apache.aurora.client # aurora # aurora-admin ``` Everything else would remain more-or-less organized the same but move to a subpackage of one of the top-level targets. The rule is simple - if both server-side and client-side import something it goes under common/, otherwise it goes under client/ or executor/ as appropriate. This seems reasonable. I'd prefer to keep the name aurora-executor but it probably merits public debate. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36972/#review93793 --- On Aug. 1, 2015, 10:38 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36972/ --- (Updated Aug. 1, 2015, 10:38 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-1381 https://issues.apache.org/jira/browse/AURORA-1381 Repository: aurora Description --- This is a mostly red code diff and makes the pants build work similar to how it already works in an IDE environment while minimizing duplicated information. This change also removes the confusing versions of `thermos` and `thermos_observer` in favor of new ones. Because of the way covering dependencies were required it was easier to do this refactor than more tactically create a working `setup_py` target for `apache.aurora.tools` without cycles and without duplicated files. * Remove the `apache.thermos` package. * Rename the `apache.gen.aurora` package to `apache.aurora.thrift`. * Rename the `apache.gen.thermos` package to `apache.thermos.thrift`. * Introduce a new `apache.aurora.tools` package with the `thermos` and `thermos_observer` binaries. * Create apache.thermos.runner package, rename `thermos/bin` to `thermos/runner`. * Remove all `*-packaged` and virtual dependency trees, as well as the phrase covering dependencies. * Use `_`-prefix naming convention for private target names. * Replace manual list of targets in `make-python-sdists` with automatically-generated one (using new convention). * Introduce a new `apache.aurora.kerberos` package with the `kaurora` and `kaurora_admin` binaries. * Remove all `BUILD` files in `src/main` that don't contain an exported `setup.py` library (except one mentioned in TODO). * Use dictionary syntax in `with_binaries`. * Remove unused/unreachable binaries (most `bin/` dirs under `thermos/`). * Create blank `__init__.py` files in directories that pants warns aren't packages. * Added `thermos` command to vagrant image. Diffs - NEWS 0a8500c524a92eaf51384125687128d2fbbb0b53 api/src/main/thrift/org/apache/aurora/gen/BUILD d196fefc2c5e5ee32d0cf9c901cffe7d247379d1 api/src/main/thrift/org/apache/thermos/BUILD d0d789a6ee3971e3070f9397d53929563a77f7ea build-support/packaging/debian/rules 17e00c02a3cb3294d5107516d795a73587ca4f70 build-support/packaging/rpm/aurora.spec 7cf8de6f7b99788ca461a90fc4aefba7dccd7b63 build-support/python/make-pycharm-virtualenv 05a16d0d421982bcfb2e649be7b83a17d813efb1 build-support/release/make-python-sdists 9608f68e16243da01434ce2fc7d61bb7c7efd712 docs/README.md 9893763cb75faf1aeb66ed905e9f696d4b532d16 docs/build-system.md PRE-CREATION examples/vagrant/aurorabuild.sh fbaa6ae9ef7ff2910af8c9c0d6b8ef90ea3e152a src/main/python/apache/aurora/admin/BUILD 22bf3f9943ea11258ba681bdb80feb00206bb926 src/main/python/apache/aurora/admin/aurora_admin.py 1c39717b656465bb1966073340f9fe80be01a085 src/main/python/apache/aurora/client/BUILD e73cd52289209bb9658b16bb77dc0b0a9c811a1a src/main/python/apache/aurora/client/api/BUILD a030a67b78fc4bc4682d0df169e27efc6810dce3
Re: Review Request 36972: One python_library per exported setup.py project
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36972/#review93932 --- Ship it! - Brian Wickman On Aug. 1, 2015, 10:38 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36972/ --- (Updated Aug. 1, 2015, 10:38 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-1381 https://issues.apache.org/jira/browse/AURORA-1381 Repository: aurora Description --- This is a mostly red code diff and makes the pants build work similar to how it already works in an IDE environment while minimizing duplicated information. This change also removes the confusing versions of `thermos` and `thermos_observer` in favor of new ones. Because of the way covering dependencies were required it was easier to do this refactor than more tactically create a working `setup_py` target for `apache.aurora.tools` without cycles and without duplicated files. * Remove the `apache.thermos` package. * Rename the `apache.gen.aurora` package to `apache.aurora.thrift`. * Rename the `apache.gen.thermos` package to `apache.thermos.thrift`. * Introduce a new `apache.aurora.tools` package with the `thermos` and `thermos_observer` binaries. * Create apache.thermos.runner package, rename `thermos/bin` to `thermos/runner`. * Remove all `*-packaged` and virtual dependency trees, as well as the phrase covering dependencies. * Use `_`-prefix naming convention for private target names. * Replace manual list of targets in `make-python-sdists` with automatically-generated one (using new convention). * Introduce a new `apache.aurora.kerberos` package with the `kaurora` and `kaurora_admin` binaries. * Remove all `BUILD` files in `src/main` that don't contain an exported `setup.py` library (except one mentioned in TODO). * Use dictionary syntax in `with_binaries`. * Remove unused/unreachable binaries (most `bin/` dirs under `thermos/`). * Create blank `__init__.py` files in directories that pants warns aren't packages. * Added `thermos` command to vagrant image. Diffs - NEWS 0a8500c524a92eaf51384125687128d2fbbb0b53 api/src/main/thrift/org/apache/aurora/gen/BUILD d196fefc2c5e5ee32d0cf9c901cffe7d247379d1 api/src/main/thrift/org/apache/thermos/BUILD d0d789a6ee3971e3070f9397d53929563a77f7ea build-support/packaging/debian/rules 17e00c02a3cb3294d5107516d795a73587ca4f70 build-support/packaging/rpm/aurora.spec 7cf8de6f7b99788ca461a90fc4aefba7dccd7b63 build-support/python/make-pycharm-virtualenv 05a16d0d421982bcfb2e649be7b83a17d813efb1 build-support/release/make-python-sdists 9608f68e16243da01434ce2fc7d61bb7c7efd712 docs/README.md 9893763cb75faf1aeb66ed905e9f696d4b532d16 docs/build-system.md PRE-CREATION examples/vagrant/aurorabuild.sh fbaa6ae9ef7ff2910af8c9c0d6b8ef90ea3e152a src/main/python/apache/aurora/admin/BUILD 22bf3f9943ea11258ba681bdb80feb00206bb926 src/main/python/apache/aurora/admin/aurora_admin.py 1c39717b656465bb1966073340f9fe80be01a085 src/main/python/apache/aurora/client/BUILD e73cd52289209bb9658b16bb77dc0b0a9c811a1a src/main/python/apache/aurora/client/api/BUILD a030a67b78fc4bc4682d0df169e27efc6810dce3 src/main/python/apache/aurora/client/cli/BUILD 973d05971bd940c7e38c48ce6bfbf5c8e1654c5a src/main/python/apache/aurora/client/cli/client.py c0974f3bef59f8f7c2320398d367cb4dd9048d2b src/main/python/apache/aurora/client/hooks/BUILD ddf813b5d2e7d63507a8e08745ebf6cb3dbac8e1 src/main/python/apache/aurora/common/BUILD abc122b0775bb17b1df67bdb946c472010219b9b src/main/python/apache/aurora/common/auth/BUILD 0abac94eeeb71e6af43ed191ea690e5f96a6be23 src/main/python/apache/aurora/common/auth/kerberos.py 2d782b63b611a9d2604ef0ab1116d3e68fb86dc7 src/main/python/apache/aurora/config/BUILD 0a3a93fa6bb785903f71bc067aecc79c0e45a0b5 src/main/python/apache/aurora/config/schema/BUILD 171f42a24ec7ddc2846eb68b6a60e7d8dec4383b src/main/python/apache/aurora/executor/BUILD 891cbe63d23ea59a68ca99f62d29be5713c9b91d src/main/python/apache/aurora/executor/bin/BUILD b23429ab9d1d3db1af2bb1b77f5bce30de19f34b src/main/python/apache/aurora/executor/common/BUILD 3d94792b9e3c2d0f7bb7ce42616f08947f5119ee src/main/python/apache/aurora/kerberos/BUILD PRE-CREATION src/main/python/apache/aurora/tools/BUILD 9dfeb6c7e5a3796c76250aa9ff6fd4230d00cfb0 src/main/python/apache/thermos/BUILD 8221aa0bd4efe5f519550cba716d6a564ba9ae44 src/main/python/apache/thermos/bin/BUILD 4f0cb9b67b1c3c52ec3278ce92ff0d765257796c src/main/python/apache/thermos/bin/__init__.py
Re: Review Request 37001: Centralizing offer/task matching in TaskAssigner.
On Aug. 3, 2015, 5:38 p.m., Zameer Manji wrote: src/main/java/org/apache/aurora/scheduler/scheduling/TaskScheduler.java, line 111 https://reviews.apache.org/r/37001/diff/1/?file=1026645#file1026645line111 nit: Was it needed to flip the BiCache type here? If not, it would be nice not to do it to minimize the diff. Swapping key/value here allows getting rid of 2 `getByValue()` calls that return set-based results. This simplifies expectations and result handling quite a bit. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37001/#review93922 --- On July 31, 2015, 11:03 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37001/ --- (Updated July 31, 2015, 11:03 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1416 https://issues.apache.org/jira/browse/AURORA-1416 Repository: aurora Description --- This releaves `OfferManager` of task matching duties and encapsulates that responsibility entirely within `TaskAssigner`. Apologies for the large diff as there is no easy way to carve out multiple commits here. The bulk of yellow/green is in test though. Summary of changes: - OfferManager becomes a purely API-based solution managing and exposing offers - TaskAssigner inherits a simplified version of OfferManager.launchFirst() and is now fully responsible for requesting a task launch - AssignmentResult is gone along with FunctionHostOffer, Assignment - TaskScheduler does not need OfferManager and StateManager anymore - TaskSchedulerTest is gone as the coverage it provides has been split between OfferManagerImplTest and TaskSchedulerImplTest. TaskGroups coverage was added in https://reviews.apache.org/r/36994 Diffs - src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 5bc73d58c5914e0ece67f2c3845c62c50ea55222 src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java f41330129b221008012fe2bd3632621badb2e125 src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 14bf265a9260548fc0a26696970fa8a84ebf687f src/main/java/org/apache/aurora/scheduler/scheduling/SchedulingModule.java c7a1a46b52cfb60d595cd31d84390e85c657cf32 src/main/java/org/apache/aurora/scheduler/scheduling/TaskScheduler.java d4bd5299ddbdf108e2520b2ab8a7517cb4c6fff9 src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 3acb45a9ffd88ccff0817acde602e133bf56ac4e src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 04be32e8bfe431f3b6a158753936e796b0cc906b src/test/java/org/apache/aurora/scheduler/scheduling/TaskSchedulerImplTest.java a2e2d4c918ad946f8bec256cb1be5e1ba8e2c40f src/test/java/org/apache/aurora/scheduler/scheduling/TaskSchedulerTest.java 9a91e63e5b7e72848ec7cac1dfadc4b6e9431216 src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java f98818fe8649e5aa0104de197bc8573feeecd37c Diff: https://reviews.apache.org/r/37001/diff/ Testing --- ./gradlew -Pq build ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Benchmark testing - no noticable difference in general apart from the limit constraint benchmark (the only dynamic group veto), which benefited from reduced complexity (no isStaticallyBanned() call on every match iteration). Before: ``` Benchmark (numPendingTasks) Mode Cnt Score Error Units SchedulingBenchmarks.ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt5 201330.070 ± 13581.745 ops/s SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt5 28745.739 ± 3323.054 ops/s SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark N/A thrpt52789.896 ± 447.990 ops/s SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark 1 thrpt5 53.487 ± 4.865 ops/s SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt5 54.061 ± 3.724 ops/s SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt5 52.667 ± 5.017 ops/s SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt5 43.241 ± 1.809 ops/s SchedulingBenchmarks.ValueConstraintMismatchSchedulingBenchmark.runBenchmark N/A thrpt5 20517.686 ± 1312.178 ops/s ```
Re: Review Request 37001: Centralizing offer/task matching in TaskAssigner.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37001/#review93922 --- Ship it! src/main/java/org/apache/aurora/scheduler/scheduling/TaskScheduler.java (line 101) https://reviews.apache.org/r/37001/#comment148359 nit: Was it needed to flip the BiCache type here? If not, it would be nice not to do it to minimize the diff. - Zameer Manji On July 31, 2015, 4:03 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37001/ --- (Updated July 31, 2015, 4:03 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1416 https://issues.apache.org/jira/browse/AURORA-1416 Repository: aurora Description --- This releaves `OfferManager` of task matching duties and encapsulates that responsibility entirely within `TaskAssigner`. Apologies for the large diff as there is no easy way to carve out multiple commits here. The bulk of yellow/green is in test though. Summary of changes: - OfferManager becomes a purely API-based solution managing and exposing offers - TaskAssigner inherits a simplified version of OfferManager.launchFirst() and is now fully responsible for requesting a task launch - AssignmentResult is gone along with FunctionHostOffer, Assignment - TaskScheduler does not need OfferManager and StateManager anymore - TaskSchedulerTest is gone as the coverage it provides has been split between OfferManagerImplTest and TaskSchedulerImplTest. TaskGroups coverage was added in https://reviews.apache.org/r/36994 Diffs - src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 5bc73d58c5914e0ece67f2c3845c62c50ea55222 src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java f41330129b221008012fe2bd3632621badb2e125 src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 14bf265a9260548fc0a26696970fa8a84ebf687f src/main/java/org/apache/aurora/scheduler/scheduling/SchedulingModule.java c7a1a46b52cfb60d595cd31d84390e85c657cf32 src/main/java/org/apache/aurora/scheduler/scheduling/TaskScheduler.java d4bd5299ddbdf108e2520b2ab8a7517cb4c6fff9 src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 3acb45a9ffd88ccff0817acde602e133bf56ac4e src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 04be32e8bfe431f3b6a158753936e796b0cc906b src/test/java/org/apache/aurora/scheduler/scheduling/TaskSchedulerImplTest.java a2e2d4c918ad946f8bec256cb1be5e1ba8e2c40f src/test/java/org/apache/aurora/scheduler/scheduling/TaskSchedulerTest.java 9a91e63e5b7e72848ec7cac1dfadc4b6e9431216 src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java f98818fe8649e5aa0104de197bc8573feeecd37c Diff: https://reviews.apache.org/r/37001/diff/ Testing --- ./gradlew -Pq build ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Benchmark testing - no noticable difference in general apart from the limit constraint benchmark (the only dynamic group veto), which benefited from reduced complexity (no isStaticallyBanned() call on every match iteration). Before: ``` Benchmark (numPendingTasks) Mode Cnt Score Error Units SchedulingBenchmarks.ClusterFullUtilizationBenchmark.runBenchmark N/A thrpt5 201330.070 ± 13581.745 ops/s SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark N/A thrpt5 28745.739 ± 3323.054 ops/s SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark N/A thrpt52789.896 ± 447.990 ops/s SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark 1 thrpt5 53.487 ± 4.865 ops/s SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark 10 thrpt5 54.061 ± 3.724 ops/s SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark 100 thrpt5 52.667 ± 5.017 ops/s SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark 1000 thrpt5 43.241 ± 1.809 ops/s SchedulingBenchmarks.ValueConstraintMismatchSchedulingBenchmark.runBenchmark N/A thrpt5 20517.686 ± 1312.178 ops/s ``` After: ``` Benchmark (numPendingTasks) Mode Cnt Score Error Units SchedulingBenchmarks.ClusterFullUtilizationBenchmark.runBenchmark
Re: Review Request 36972: One python_library per exported setup.py project
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36972/#review93930 --- Ship it! Ship It! - Bill Farner On Aug. 1, 2015, 10:38 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36972/ --- (Updated Aug. 1, 2015, 10:38 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-1381 https://issues.apache.org/jira/browse/AURORA-1381 Repository: aurora Description --- This is a mostly red code diff and makes the pants build work similar to how it already works in an IDE environment while minimizing duplicated information. This change also removes the confusing versions of `thermos` and `thermos_observer` in favor of new ones. Because of the way covering dependencies were required it was easier to do this refactor than more tactically create a working `setup_py` target for `apache.aurora.tools` without cycles and without duplicated files. * Remove the `apache.thermos` package. * Rename the `apache.gen.aurora` package to `apache.aurora.thrift`. * Rename the `apache.gen.thermos` package to `apache.thermos.thrift`. * Introduce a new `apache.aurora.tools` package with the `thermos` and `thermos_observer` binaries. * Create apache.thermos.runner package, rename `thermos/bin` to `thermos/runner`. * Remove all `*-packaged` and virtual dependency trees, as well as the phrase covering dependencies. * Use `_`-prefix naming convention for private target names. * Replace manual list of targets in `make-python-sdists` with automatically-generated one (using new convention). * Introduce a new `apache.aurora.kerberos` package with the `kaurora` and `kaurora_admin` binaries. * Remove all `BUILD` files in `src/main` that don't contain an exported `setup.py` library (except one mentioned in TODO). * Use dictionary syntax in `with_binaries`. * Remove unused/unreachable binaries (most `bin/` dirs under `thermos/`). * Create blank `__init__.py` files in directories that pants warns aren't packages. * Added `thermos` command to vagrant image. Diffs - NEWS 0a8500c524a92eaf51384125687128d2fbbb0b53 api/src/main/thrift/org/apache/aurora/gen/BUILD d196fefc2c5e5ee32d0cf9c901cffe7d247379d1 api/src/main/thrift/org/apache/thermos/BUILD d0d789a6ee3971e3070f9397d53929563a77f7ea build-support/packaging/debian/rules 17e00c02a3cb3294d5107516d795a73587ca4f70 build-support/packaging/rpm/aurora.spec 7cf8de6f7b99788ca461a90fc4aefba7dccd7b63 build-support/python/make-pycharm-virtualenv 05a16d0d421982bcfb2e649be7b83a17d813efb1 build-support/release/make-python-sdists 9608f68e16243da01434ce2fc7d61bb7c7efd712 docs/README.md 9893763cb75faf1aeb66ed905e9f696d4b532d16 docs/build-system.md PRE-CREATION examples/vagrant/aurorabuild.sh fbaa6ae9ef7ff2910af8c9c0d6b8ef90ea3e152a src/main/python/apache/aurora/admin/BUILD 22bf3f9943ea11258ba681bdb80feb00206bb926 src/main/python/apache/aurora/admin/aurora_admin.py 1c39717b656465bb1966073340f9fe80be01a085 src/main/python/apache/aurora/client/BUILD e73cd52289209bb9658b16bb77dc0b0a9c811a1a src/main/python/apache/aurora/client/api/BUILD a030a67b78fc4bc4682d0df169e27efc6810dce3 src/main/python/apache/aurora/client/cli/BUILD 973d05971bd940c7e38c48ce6bfbf5c8e1654c5a src/main/python/apache/aurora/client/cli/client.py c0974f3bef59f8f7c2320398d367cb4dd9048d2b src/main/python/apache/aurora/client/hooks/BUILD ddf813b5d2e7d63507a8e08745ebf6cb3dbac8e1 src/main/python/apache/aurora/common/BUILD abc122b0775bb17b1df67bdb946c472010219b9b src/main/python/apache/aurora/common/auth/BUILD 0abac94eeeb71e6af43ed191ea690e5f96a6be23 src/main/python/apache/aurora/common/auth/kerberos.py 2d782b63b611a9d2604ef0ab1116d3e68fb86dc7 src/main/python/apache/aurora/config/BUILD 0a3a93fa6bb785903f71bc067aecc79c0e45a0b5 src/main/python/apache/aurora/config/schema/BUILD 171f42a24ec7ddc2846eb68b6a60e7d8dec4383b src/main/python/apache/aurora/executor/BUILD 891cbe63d23ea59a68ca99f62d29be5713c9b91d src/main/python/apache/aurora/executor/bin/BUILD b23429ab9d1d3db1af2bb1b77f5bce30de19f34b src/main/python/apache/aurora/executor/common/BUILD 3d94792b9e3c2d0f7bb7ce42616f08947f5119ee src/main/python/apache/aurora/kerberos/BUILD PRE-CREATION src/main/python/apache/aurora/tools/BUILD 9dfeb6c7e5a3796c76250aa9ff6fd4230d00cfb0 src/main/python/apache/thermos/BUILD 8221aa0bd4efe5f519550cba716d6a564ba9ae44 src/main/python/apache/thermos/bin/BUILD 4f0cb9b67b1c3c52ec3278ce92ff0d765257796c src/main/python/apache/thermos/bin/__init__.py
