Re: Review Request 33106: Simplify AttributeAggregate.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33106/#review80047 --- Ship it! Ship It! - Maxim Khutornenko On April 14, 2015, 4 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33106/ --- (Updated April 14, 2015, 4 a.m.) Review request for Aurora and Maxim Khutornenko. Repository: aurora Description --- Use `Multiset` instead of `AtomicLongMap`, introduce a convenience constant for unit tests to use. Diffs - src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java ce87344dc18818faa7a1a0298143dc81fff7 src/main/java/org/apache/aurora/scheduler/filter/AttributeAggregate.java ed82ae99f23d5a7f1634261205cbe5339fe4cec8 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerImplTest.java c5643d9d99fc46d55fd6c48161230139fb7f12b8 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 88c0163170ebc25995d9ef8b1543335a4322bb8e src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java bcd1b4e854f5ea227268c73156bc97c7c937c1de src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 281f4e02650727aa5d0a35a09dcf0eb823ad1b50 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorModuleTest.java 7e2d1c54362b33cc3507a4bc3e3ccc02ca29bd6f src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorSlotFinderTest.java b80e558f18b817814e4768b13ff94aa816d28543 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 61cea326ababcd6242a3c5a6dcf8d0b3ca7fbdd6 src/test/java/org/apache/aurora/scheduler/filter/AttributeAggregateTest.java 4b565769d79862326efcb31be694f95f333c89c6 src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java d06b89cc319aa7b4479124cbb2cb224cdb662e05 src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java aca0234e037e85202d182affa2c0e988c6cfc854 Diff: https://reviews.apache.org/r/33106/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 32907: Generalizing preemption reservation pool.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32907/#review80052 --- Ping Zameer. - Maxim Khutornenko On April 14, 2015, 1:09 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32907/ --- (Updated April 14, 2015, 1:09 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1219 https://issues.apache.org/jira/browse/AURORA-1219 Repository: aurora Description --- Current preemption model assumes tracking preemption slots/reservations by taskId. This reduces preemption efficiency as it only takes a specific taskId scheduling round to claim the PreemptionSlot and then subsequently make a slave reservation. This diff generalizes the preemption pool by TaskGroupKey. Preemption slots are now created and cached by TaskGroupKey and are available to _any_ task from the same TaskGroup. This change should also simplify the algorithm in https://reviews.apache.org/r/32597/. Diffs - src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java e87dda47a355654c66f6f54fb25a4d9a7f68422d src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ebc520ebb4dddbc69b2b4a6f9174c1451d61887a src/main/java/org/apache/aurora/scheduler/async/preemptor/BiCache.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessor.java 67ad5d7f9909bc892301c19586561b6cdbfe79e6 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCache.java b5a42a0a0498f06a382576e2c3a839bbb7f1d2b1 src/main/java/org/apache/aurora/scheduler/async/preemptor/Preemptor.java 77617ec9a2bcd062d5b80cd2b4c58dc80514 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorModule.java 1092c055588363794b37a965fb2f17a6e50d92d7 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerImplTest.java c5643d9d99fc46d55fd6c48161230139fb7f12b8 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 88c0163170ebc25995d9ef8b1543335a4322bb8e src/test/java/org/apache/aurora/scheduler/async/preemptor/BiCacheTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java bcd1b4e854f5ea227268c73156bc97c7c937c1de src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCacheTest.java 80bd13a192bda64759b5a93749ec47ddfeae504a src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 281f4e02650727aa5d0a35a09dcf0eb823ad1b50 Diff: https://reviews.apache.org/r/32907/diff/ Testing --- ./gradlew -Pq build Manual testing in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 32907: Generalizing preemption reservation pool.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32907/ --- (Updated April 14, 2015, 6:17 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Changes --- Rebased. Bugs: AURORA-1219 https://issues.apache.org/jira/browse/AURORA-1219 Repository: aurora Description --- Current preemption model assumes tracking preemption slots/reservations by taskId. This reduces preemption efficiency as it only takes a specific taskId scheduling round to claim the PreemptionSlot and then subsequently make a slave reservation. This diff generalizes the preemption pool by TaskGroupKey. Preemption slots are now created and cached by TaskGroupKey and are available to _any_ task from the same TaskGroup. This change should also simplify the algorithm in https://reviews.apache.org/r/32597/. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java e87dda47a355654c66f6f54fb25a4d9a7f68422d src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ebc520ebb4dddbc69b2b4a6f9174c1451d61887a src/main/java/org/apache/aurora/scheduler/async/preemptor/BiCache.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessor.java 67ad5d7f9909bc892301c19586561b6cdbfe79e6 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCache.java b5a42a0a0498f06a382576e2c3a839bbb7f1d2b1 src/main/java/org/apache/aurora/scheduler/async/preemptor/Preemptor.java 77617ec9a2bcd062d5b80cd2b4c58dc80514 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorModule.java 1092c055588363794b37a965fb2f17a6e50d92d7 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerImplTest.java b61abf93eb6bb7dee336a675546bc604612b130e src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 9c47a768ec9e5c527e7f2c50332ef77433c8e088 src/test/java/org/apache/aurora/scheduler/async/preemptor/BiCacheTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java 75fc16dd2796ef2d5eebea6575d141cf2a4fade0 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCacheTest.java 80bd13a192bda64759b5a93749ec47ddfeae504a src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 97d6087ab92ae6d492f50ccf6c6091ec42b1c032 Diff: https://reviews.apache.org/r/32907/diff/ Testing --- ./gradlew -Pq build Manual testing in vagrant. Thanks, Maxim Khutornenko
Review Request 33202: Fixing benchmarks.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33202/ --- Review request for Aurora and Bill Farner. Repository: aurora Description --- Benchmark module configuration got broken by recent refactoring. Diffs - src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 011350540d4f5091ff5f8a3c646f40f3f63357fd Diff: https://reviews.apache.org/r/33202/diff/ Testing --- ./gradlew jmh Thanks, Maxim Khutornenko
Re: Review Request 33202: Fixing benchmarks.
On April 15, 2015, 3:07 a.m., Bill Farner wrote: src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java, line 135 https://reviews.apache.org/r/33202/diff/1/?file=929440#file929440line135 IMHO the empty string is a wart, and emphasizes my point about letting the consumer decide if they want stats. I won't harp on it, but i think this has made the code less reusable. Maxim Khutornenko wrote: This is hardly a good example though as we always want stats in production. I will be happy to reconsider should we have a real counter case. So far, every time we deal with caches we always want more visibility. This probably makes a good case for a decorating implementation of an interface. No urgency, of course...but something to keep in mind as things evolve. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33202/#review80148 --- On April 15, 2015, 12:32 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33202/ --- (Updated April 15, 2015, 12:32 a.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- Benchmark module configuration got broken by recent refactoring. Diffs - src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 011350540d4f5091ff5f8a3c646f40f3f63357fd Diff: https://reviews.apache.org/r/33202/diff/ Testing --- ./gradlew jmh Thanks, Maxim Khutornenko
Re: Review Request 32907: Generalizing preemption reservation pool.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32907/#review80076 --- Ship it! Master (3b29a4b) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 14, 2015, 6:17 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32907/ --- (Updated April 14, 2015, 6:17 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1219 https://issues.apache.org/jira/browse/AURORA-1219 Repository: aurora Description --- Current preemption model assumes tracking preemption slots/reservations by taskId. This reduces preemption efficiency as it only takes a specific taskId scheduling round to claim the PreemptionSlot and then subsequently make a slave reservation. This diff generalizes the preemption pool by TaskGroupKey. Preemption slots are now created and cached by TaskGroupKey and are available to _any_ task from the same TaskGroup. This change should also simplify the algorithm in https://reviews.apache.org/r/32597/. Diffs - src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java e87dda47a355654c66f6f54fb25a4d9a7f68422d src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ebc520ebb4dddbc69b2b4a6f9174c1451d61887a src/main/java/org/apache/aurora/scheduler/async/preemptor/BiCache.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessor.java 67ad5d7f9909bc892301c19586561b6cdbfe79e6 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCache.java b5a42a0a0498f06a382576e2c3a839bbb7f1d2b1 src/main/java/org/apache/aurora/scheduler/async/preemptor/Preemptor.java 77617ec9a2bcd062d5b80cd2b4c58dc80514 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorModule.java 1092c055588363794b37a965fb2f17a6e50d92d7 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerImplTest.java b61abf93eb6bb7dee336a675546bc604612b130e src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 9c47a768ec9e5c527e7f2c50332ef77433c8e088 src/test/java/org/apache/aurora/scheduler/async/preemptor/BiCacheTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java 75fc16dd2796ef2d5eebea6575d141cf2a4fade0 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCacheTest.java 80bd13a192bda64759b5a93749ec47ddfeae504a src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 97d6087ab92ae6d492f50ccf6c6091ec42b1c032 Diff: https://reviews.apache.org/r/32907/diff/ Testing --- ./gradlew -Pq build Manual testing in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 33105: Return Iterable from TaskStore.fetchTasks to allow for streaming.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33105/#review80123 --- Ship it! Master (1dc11fb) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 14, 2015, 9:29 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33105/ --- (Updated April 14, 2015, 9:29 p.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Repository: aurora Description --- This allows the store implementation to stream if it wishes, which we might want to take advantage of when the store is backed by a database. One thing worth noting is that this change is a good example of why using wide types at call sites is important. There were many callers of TaskStore.fetchTasks that were already assigning as `Iterable`, and they were immune to this refactor. Diffs - src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 1da35c05fb22c8fa0227a91df6a19f1b8f1aae92 src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 3bb80ec99ed1f5c1d87edb59f3c9502ab6f7c706 src/main/java/org/apache/aurora/scheduler/async/RescheduleCalculator.java 0cf7fb40e5640d0ad07a7fc46c57548a3a385872 src/main/java/org/apache/aurora/scheduler/async/TaskHistoryPruner.java 985a3196ad809c7ccf30ba87a1f10cc3dacd5f49 src/main/java/org/apache/aurora/scheduler/cron/quartz/AuroraCronJob.java 3b5dcf819228297b3840cb01788c3085759f3c4a src/main/java/org/apache/aurora/scheduler/state/MaintenanceController.java 09be4dc8d6318ccfdf10397585ef7989221afd7f src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java b6a7b4aca841025c790f4b282f0490ff5bb23b2f src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java a8e3b14b3f8147f60871f9dfa84616425636e10a src/main/java/org/apache/aurora/scheduler/storage/Storage.java 6180a36255253542197234ec610eea01366135bc src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java b76c937281971a505db4b352b80cbb7ccfcafcca src/main/java/org/apache/aurora/scheduler/storage/backup/Recovery.java 38764e5e7484e2fe152460ac7920be9a5799b85d src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java 2102adb3844fc1501ff9d672a2db78a69513231b src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 7aef1ca786fea521256f69f6a9e7a978649a7195 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java b7d3874e9f597d93c9f623773a3a0289c2b76d6d src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 34cbd198de8e3689b77232f9ca78a2bbb6023106 Diff: https://reviews.apache.org/r/33105/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 33184: Implement missing != operator for AuroraJobKey
On April 14, 2015, 8:53 p.m., Bill Farner wrote: src/test/python/apache/aurora/common/test_aurora_job_key.py, line 25 https://reviews.apache.org/r/33184/diff/1/?file=927354#file927354line25 Can you test the inverse as well - keys that are not equal? Added. It is a little bit longer as we have to test all combinations. - Stephan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33184/#review80079 --- On April 14, 2015, 9:18 p.m., Stephan Erb wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33184/ --- (Updated April 14, 2015, 9:18 p.m.) Review request for Aurora. Repository: aurora Description --- Implement missing != operator for AuroraJobKey From the Python docu of __ne__: There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false. Accordingly, when defining __eq__(), one should also define __ne__() so that the operators will behave as expected. Diffs - src/main/python/apache/aurora/common/aurora_job_key.py 88896c6c41235f7d313f0d77682acdca7cce src/test/python/apache/aurora/common/test_aurora_job_key.py 7d8d58b541f16d63495f46b58bd928e8f53f66d1 Diff: https://reviews.apache.org/r/33184/diff/ Testing --- ./pants test.pytest --no-fast --options=-v src/test/python/apache/aurora/common:: Thanks, Stephan Erb
Re: Review Request 33184: Implement missing != operator for AuroraJobKey
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33184/#review80079 --- src/test/python/apache/aurora/common/test_aurora_job_key.py https://reviews.apache.org/r/33184/#comment129848 Can you test the inverse as well - keys that are not equal? - Bill Farner On April 14, 2015, 6:51 p.m., Stephan Erb wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33184/ --- (Updated April 14, 2015, 6:51 p.m.) Review request for Aurora. Repository: aurora Description --- Implement missing \!= operator for AuroraJobKey From the Python docu of __ne__: There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false. Accordingly, when defining __eq__(), one should also define __ne__() so that the operators will behave as expected. Diffs - src/main/python/apache/aurora/common/aurora_job_key.py 88896c6c41235f7d313f0d77682acdca7cce src/test/python/apache/aurora/common/test_aurora_job_key.py 7d8d58b541f16d63495f46b58bd928e8f53f66d1 Diff: https://reviews.apache.org/r/33184/diff/ Testing --- ./pants test.pytest --no-fast --options=-v src/test/python/apache/aurora/common:: Thanks, Stephan Erb
Re: Review Request 33184: Implement missing != operator for AuroraJobKey
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33184/ --- (Updated April 14, 2015, 8:54 p.m.) Review request for Aurora. Repository: aurora Description (updated) --- Implement missing != operator for AuroraJobKey From the Python docu of __ne__: There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false. Accordingly, when defining __eq__(), one should also define __ne__() so that the operators will behave as expected. Diffs - src/main/python/apache/aurora/common/aurora_job_key.py 88896c6c41235f7d313f0d77682acdca7cce src/test/python/apache/aurora/common/test_aurora_job_key.py 7d8d58b541f16d63495f46b58bd928e8f53f66d1 Diff: https://reviews.apache.org/r/33184/diff/ Testing --- ./pants test.pytest --no-fast --options=-v src/test/python/apache/aurora/common:: Thanks, Stephan Erb
Re: Review Request 32907: Generalizing preemption reservation pool.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32907/ --- (Updated April 14, 2015, 8 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Changes --- Zameer's comments. Bugs: AURORA-1219 https://issues.apache.org/jira/browse/AURORA-1219 Repository: aurora Description --- Current preemption model assumes tracking preemption slots/reservations by taskId. This reduces preemption efficiency as it only takes a specific taskId scheduling round to claim the PreemptionSlot and then subsequently make a slave reservation. This diff generalizes the preemption pool by TaskGroupKey. Preemption slots are now created and cached by TaskGroupKey and are available to _any_ task from the same TaskGroup. This change should also simplify the algorithm in https://reviews.apache.org/r/32597/. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java e87dda47a355654c66f6f54fb25a4d9a7f68422d src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ebc520ebb4dddbc69b2b4a6f9174c1451d61887a src/main/java/org/apache/aurora/scheduler/async/preemptor/BiCache.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessor.java 67ad5d7f9909bc892301c19586561b6cdbfe79e6 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCache.java b5a42a0a0498f06a382576e2c3a839bbb7f1d2b1 src/main/java/org/apache/aurora/scheduler/async/preemptor/Preemptor.java 77617ec9a2bcd062d5b80cd2b4c58dc80514 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorModule.java 1092c055588363794b37a965fb2f17a6e50d92d7 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerImplTest.java b61abf93eb6bb7dee336a675546bc604612b130e src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 9c47a768ec9e5c527e7f2c50332ef77433c8e088 src/test/java/org/apache/aurora/scheduler/async/preemptor/BiCacheTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java 75fc16dd2796ef2d5eebea6575d141cf2a4fade0 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCacheTest.java 80bd13a192bda64759b5a93749ec47ddfeae504a src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 97d6087ab92ae6d492f50ccf6c6091ec42b1c032 Diff: https://reviews.apache.org/r/32907/diff/ Testing --- ./gradlew -Pq build Manual testing in vagrant. Thanks, Maxim Khutornenko
