Re: Review Request 33105: Return Iterable from TaskStore.fetchTasks to allow for streaming.

2015-04-14 Thread Aurora ReviewBot

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33105/#review80123
---

Ship it!


Master (1dc11fb) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing @ReviewBot 
retry

- Aurora ReviewBot


On April 14, 2015, 9:29 p.m., Bill Farner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33105/
 ---
 
 (Updated April 14, 2015, 9:29 p.m.)
 
 
 Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
 
 
 Repository: aurora
 
 
 Description
 ---
 
 This allows the store implementation to stream if it wishes, which we might 
 want to take advantage of when the store is backed by a database.
 
 One thing worth noting is that this change is a good example of why using 
 wide types at call sites is important.  There were many callers of 
 TaskStore.fetchTasks that were already assigning as `Iterable`, and they were 
 immune to this refactor.
 
 
 Diffs
 -
 
   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
 1da35c05fb22c8fa0227a91df6a19f1b8f1aae92 
   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
 3bb80ec99ed1f5c1d87edb59f3c9502ab6f7c706 
   src/main/java/org/apache/aurora/scheduler/async/RescheduleCalculator.java 
 0cf7fb40e5640d0ad07a7fc46c57548a3a385872 
   src/main/java/org/apache/aurora/scheduler/async/TaskHistoryPruner.java 
 985a3196ad809c7ccf30ba87a1f10cc3dacd5f49 
   src/main/java/org/apache/aurora/scheduler/cron/quartz/AuroraCronJob.java 
 3b5dcf819228297b3840cb01788c3085759f3c4a 
   src/main/java/org/apache/aurora/scheduler/state/MaintenanceController.java 
 09be4dc8d6318ccfdf10397585ef7989221afd7f 
   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
 b6a7b4aca841025c790f4b282f0490ff5bb23b2f 
   src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java 
 a8e3b14b3f8147f60871f9dfa84616425636e10a 
   src/main/java/org/apache/aurora/scheduler/storage/Storage.java 
 6180a36255253542197234ec610eea01366135bc 
   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
 b76c937281971a505db4b352b80cbb7ccfcafcca 
   src/main/java/org/apache/aurora/scheduler/storage/backup/Recovery.java 
 38764e5e7484e2fe152460ac7920be9a5799b85d 
   
 src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java
  2102adb3844fc1501ff9d672a2db78a69513231b 
   src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
 7aef1ca786fea521256f69f6a9e7a978649a7195 
   
 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
  b7d3874e9f597d93c9f623773a3a0289c2b76d6d 
   src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 
 34cbd198de8e3689b77232f9ca78a2bbb6023106 
 
 Diff: https://reviews.apache.org/r/33105/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Bill Farner
 




Re: Review Request 33105: Return Iterable from TaskStore.fetchTasks to allow for streaming.

2015-04-13 Thread Maxim Khutornenko

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33105/#review79960
---

Ship it!


Ship It!

- Maxim Khutornenko


On April 11, 2015, 6:22 p.m., Bill Farner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33105/
 ---
 
 (Updated April 11, 2015, 6:22 p.m.)
 
 
 Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
 
 
 Repository: aurora
 
 
 Description
 ---
 
 This allows the store implementation to stream if it wishes, which we might 
 want to take advantage of when the store is backed by a database.
 
 One thing worth noting is that this change is a good example of why using 
 wide types at call sites is important.  There were many callers of 
 TaskStore.fetchTasks that were already assigning as `Iterable`, and they were 
 immune to this refactor.
 
 
 Diffs
 -
 
   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
 ce87344dc18818faa7a1a0298143dc81fff7 
   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
 1da35c05fb22c8fa0227a91df6a19f1b8f1aae92 
   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
 3bb80ec99ed1f5c1d87edb59f3c9502ab6f7c706 
   src/main/java/org/apache/aurora/scheduler/async/RescheduleCalculator.java 
 0cf7fb40e5640d0ad07a7fc46c57548a3a385872 
   src/main/java/org/apache/aurora/scheduler/async/TaskHistoryPruner.java 
 985a3196ad809c7ccf30ba87a1f10cc3dacd5f49 
   src/main/java/org/apache/aurora/scheduler/cron/quartz/AuroraCronJob.java 
 3b5dcf819228297b3840cb01788c3085759f3c4a 
   src/main/java/org/apache/aurora/scheduler/filter/AttributeAggregate.java 
 ed82ae99f23d5a7f1634261205cbe5339fe4cec8 
   src/main/java/org/apache/aurora/scheduler/state/MaintenanceController.java 
 09be4dc8d6318ccfdf10397585ef7989221afd7f 
   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
 b6a7b4aca841025c790f4b282f0490ff5bb23b2f 
   src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java 
 a8e3b14b3f8147f60871f9dfa84616425636e10a 
   src/main/java/org/apache/aurora/scheduler/storage/Storage.java 
 6180a36255253542197234ec610eea01366135bc 
   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
 b76c937281971a505db4b352b80cbb7ccfcafcca 
   src/main/java/org/apache/aurora/scheduler/storage/backup/Recovery.java 
 38764e5e7484e2fe152460ac7920be9a5799b85d 
   
 src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java
  2102adb3844fc1501ff9d672a2db78a69513231b 
   src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
 7aef1ca786fea521256f69f6a9e7a978649a7195 
   
 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
  b7d3874e9f597d93c9f623773a3a0289c2b76d6d 
   src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerImplTest.java 
 c5643d9d99fc46d55fd6c48161230139fb7f12b8 
   src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 
 88c0163170ebc25995d9ef8b1543335a4322bb8e 
   
 src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java
  bcd1b4e854f5ea227268c73156bc97c7c937c1de 
   
 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java
  281f4e02650727aa5d0a35a09dcf0eb823ad1b50 
   
 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorModuleTest.java
  7e2d1c54362b33cc3507a4bc3e3ccc02ca29bd6f 
   
 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorSlotFinderTest.java
  b80e558f18b817814e4768b13ff94aa816d28543 
   
 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java
  61cea326ababcd6242a3c5a6dcf8d0b3ca7fbdd6 
   
 src/test/java/org/apache/aurora/scheduler/filter/AttributeAggregateTest.java 
 4b565769d79862326efcb31be694f95f333c89c6 
   
 src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java
  d06b89cc319aa7b4479124cbb2cb224cdb662e05 
   src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
 aca0234e037e85202d182affa2c0e988c6cfc854 
 
 Diff: https://reviews.apache.org/r/33105/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Bill Farner
 




Re: Review Request 33105: Return Iterable from TaskStore.fetchTasks to allow for streaming.

2015-04-13 Thread Bill Farner


 On April 13, 2015, 11:06 p.m., Kevin Sweeney wrote:
  src/main/java/org/apache/aurora/scheduler/async/RescheduleCalculator.java, 
  lines 149-152
  https://reviews.apache.org/r/33105/diff/1/?file=924537#file924537line149
 
  Inline this as FluentIterable.from(...).first()

`first()` changes behavior in that it will succeed if there are multiple items 
in the `Iterable`.  Are you comfortable with that?


 On April 13, 2015, 11:06 p.m., Kevin Sweeney wrote:
  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java, line 37
  https://reviews.apache.org/r/33105/diff/1/?file=924545#file924545line37
 
  Auditing the callers it looks like we might also want a 
  `OptionalIScheduledTask fetchTask(String taskId);`.

There are several API-level changes to be made.  For now i would like to start 
with pure API compatibility to avoid pulling any more on a very long thread.


 On April 13, 2015, 11:06 p.m., Kevin Sweeney wrote:
  src/main/java/org/apache/aurora/scheduler/async/TaskHistoryPruner.java, 
  line 160
  https://reviews.apache.org/r/33105/diff/1/?file=924538#file924538line160
 
  This is potentially very expensive to this caller if the underlying 
  implementation is not a collection - drop a TODO to expose a COUNT() 
  interface?

I agree.  My goal for right now is to minimally change APIs and re-assess 
callers for needed functions a bit further down the road.  I'd like to avoid 
dropping TODOs around, as performance will become the forcing function for 
these changes.


- Bill


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33105/#review79962
---


On April 11, 2015, 6:22 p.m., Bill Farner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33105/
 ---
 
 (Updated April 11, 2015, 6:22 p.m.)
 
 
 Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
 
 
 Repository: aurora
 
 
 Description
 ---
 
 This allows the store implementation to stream if it wishes, which we might 
 want to take advantage of when the store is backed by a database.
 
 One thing worth noting is that this change is a good example of why using 
 wide types at call sites is important.  There were many callers of 
 TaskStore.fetchTasks that were already assigning as `Iterable`, and they were 
 immune to this refactor.
 
 
 Diffs
 -
 
   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
 ce87344dc18818faa7a1a0298143dc81fff7 
   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
 1da35c05fb22c8fa0227a91df6a19f1b8f1aae92 
   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
 3bb80ec99ed1f5c1d87edb59f3c9502ab6f7c706 
   src/main/java/org/apache/aurora/scheduler/async/RescheduleCalculator.java 
 0cf7fb40e5640d0ad07a7fc46c57548a3a385872 
   src/main/java/org/apache/aurora/scheduler/async/TaskHistoryPruner.java 
 985a3196ad809c7ccf30ba87a1f10cc3dacd5f49 
   src/main/java/org/apache/aurora/scheduler/cron/quartz/AuroraCronJob.java 
 3b5dcf819228297b3840cb01788c3085759f3c4a 
   src/main/java/org/apache/aurora/scheduler/filter/AttributeAggregate.java 
 ed82ae99f23d5a7f1634261205cbe5339fe4cec8 
   src/main/java/org/apache/aurora/scheduler/state/MaintenanceController.java 
 09be4dc8d6318ccfdf10397585ef7989221afd7f 
   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
 b6a7b4aca841025c790f4b282f0490ff5bb23b2f 
   src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java 
 a8e3b14b3f8147f60871f9dfa84616425636e10a 
   src/main/java/org/apache/aurora/scheduler/storage/Storage.java 
 6180a36255253542197234ec610eea01366135bc 
   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
 b76c937281971a505db4b352b80cbb7ccfcafcca 
   src/main/java/org/apache/aurora/scheduler/storage/backup/Recovery.java 
 38764e5e7484e2fe152460ac7920be9a5799b85d 
   
 src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java
  2102adb3844fc1501ff9d672a2db78a69513231b 
   src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
 7aef1ca786fea521256f69f6a9e7a978649a7195 
   
 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
  b7d3874e9f597d93c9f623773a3a0289c2b76d6d 
   src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerImplTest.java 
 c5643d9d99fc46d55fd6c48161230139fb7f12b8 
   src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 
 88c0163170ebc25995d9ef8b1543335a4322bb8e 
   
 src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java
  bcd1b4e854f5ea227268c73156bc97c7c937c1de 
   
 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java
  

Re: Review Request 33105: Return Iterable from TaskStore.fetchTasks to allow for streaming.

2015-04-11 Thread Aurora ReviewBot

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33105/#review79809
---


Master (eb9e408) is red with this patch.
  ./build-support/jenkins/build.sh

:check
:build
:api:assemble
:api:compileTestJava UP-TO-DATE
:api:processTestResources UP-TO-DATE
:api:testClasses UP-TO-DATE
:api:test UP-TO-DATE
:api:check UP-TO-DATE
:api:build
:buildSrc:compileJava UP-TO-DATE
:buildSrc:processResources UP-TO-DATE
:buildSrc:classes UP-TO-DATE
:buildSrc:jar
:buildSrc:assemble
:buildSrc:compileTestJava UP-TO-DATE
:buildSrc:processTestResources UP-TO-DATE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test UP-TO-DATE
:buildSrc:check UP-TO-DATE
:buildSrc:build

BUILD SUCCESSFUL

Total time: 4 mins 7.27 secs
+ export PIP_DEFAULT_TIMEOUT=60
+ PIP_DEFAULT_TIMEOUT=60
+ mkdir -p third_party
+ pip install -d third_party -r /dev/fd/63
++ grep -v mesos.native 3rdparty/python/requirements.txt
Downloading/unpacking bottle==0.11.6 (from -r /dev/fd/63 (line 1))
  Saved ./third_party/bottle-0.11.6-py2.py3-none-any.whl
Downloading/unpacking CherryPy==3.6.0 (from -r /dev/fd/63 (line 2))

pip can't proceed with requirement 'CherryPy==3.6.0 (from -r /dev/fd/63 (line 
2))' due to a pre-existing build directory.
 location: /tmp/user/2395/pip_build_jenkins/CherryPy
This is likely due to a previous installation that failed.
pip is being responsible and not assuming it can delete this.
Please delete it and try again.

Storing debug log for failure in /home/jenkins/.pip/pip.log


I will refresh this build result if you post a review containing @ReviewBot 
retry

- Aurora ReviewBot


On April 11, 2015, 6:22 p.m., Bill Farner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/33105/
 ---
 
 (Updated April 11, 2015, 6:22 p.m.)
 
 
 Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
 
 
 Repository: aurora
 
 
 Description
 ---
 
 This allows the store implementation to stream if it wishes, which we might 
 want to take advantage of when the store is backed by a database.
 
 One thing worth noting is that this change is a good example of why using 
 wide types at call sites is important.  There were many callers of 
 TaskStore.fetchTasks that were already assigning as `Iterable`, and they were 
 immune to this refactor.
 
 
 Diffs
 -
 
   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
 ce87344dc18818faa7a1a0298143dc81fff7 
   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
 1da35c05fb22c8fa0227a91df6a19f1b8f1aae92 
   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
 3bb80ec99ed1f5c1d87edb59f3c9502ab6f7c706 
   src/main/java/org/apache/aurora/scheduler/async/RescheduleCalculator.java 
 0cf7fb40e5640d0ad07a7fc46c57548a3a385872 
   src/main/java/org/apache/aurora/scheduler/async/TaskHistoryPruner.java 
 985a3196ad809c7ccf30ba87a1f10cc3dacd5f49 
   src/main/java/org/apache/aurora/scheduler/cron/quartz/AuroraCronJob.java 
 3b5dcf819228297b3840cb01788c3085759f3c4a 
   src/main/java/org/apache/aurora/scheduler/filter/AttributeAggregate.java 
 ed82ae99f23d5a7f1634261205cbe5339fe4cec8 
   src/main/java/org/apache/aurora/scheduler/state/MaintenanceController.java 
 09be4dc8d6318ccfdf10397585ef7989221afd7f 
   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
 b6a7b4aca841025c790f4b282f0490ff5bb23b2f 
   src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java 
 a8e3b14b3f8147f60871f9dfa84616425636e10a 
   src/main/java/org/apache/aurora/scheduler/storage/Storage.java 
 6180a36255253542197234ec610eea01366135bc 
   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
 b76c937281971a505db4b352b80cbb7ccfcafcca 
   src/main/java/org/apache/aurora/scheduler/storage/backup/Recovery.java 
 38764e5e7484e2fe152460ac7920be9a5799b85d 
   
 src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java
  2102adb3844fc1501ff9d672a2db78a69513231b 
   src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
 7aef1ca786fea521256f69f6a9e7a978649a7195 
   
 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
  b7d3874e9f597d93c9f623773a3a0289c2b76d6d 
   src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerImplTest.java 
 c5643d9d99fc46d55fd6c48161230139fb7f12b8 
   src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 
 88c0163170ebc25995d9ef8b1543335a4322bb8e 
   
 src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java
  bcd1b4e854f5ea227268c73156bc97c7c937c1de 
   
 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java
  281f4e02650727aa5d0a35a09dcf0eb823ad1b50