Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Bill Farner

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/
---

Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.


Bugs: AURORA-920
https://issues.apache.org/jira/browse/AURORA-920


Repository: aurora


Description
---

Apologies for the large diff - this is a thread that began with fixing a 
SchedulerDriver-related race that would exist if we moved to an async event 
bus.  Most of the green code is not new, but extracted from broken-apart and/or 
deleted files.  However, i think this refactor makes the lifecycle of the 
scheduler (and driver) easier to reason about.

Changes of note:
- We now use AbstractIdleService to manage the call order on the 
SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
- Many mesos-specific classes were moved to `org.apache.aurora.scheduler.mesos`
- Command line args used directly from `DriverFactoryImpl` are now properly 
injected via `DriverSettings`
- Untestable code around creating the driver is cordoned off to 
`DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
constructor
- Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
and `CommandLineDriverSettingsModule`


Diffs
-

  config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
  src/main/java/org/apache/aurora/scheduler/Driver.java 
b07378f3772136177c81a65497152e9d6524b0a7 
  src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
9cc04a84a37374ffca418e2ff767992ee23b9f3e 
  src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
  src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
  src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
453f22a8e97dbf535dd99598332e71e944ebf8d2 
  src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
8d306409bc382fbe5c53bb394d03c923b8f75889 
  src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
9ec3f412a9a9210279a2ce666bd99f61481ab92d 
  src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
048740953629a950c136179c9b880b8dce8fa932 
  src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
  src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
feb70a9351fe804b6745304e4cbbf6d010c56579 
  src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
0b023a25292907b8211274f1533d89de7c4b7ce4 
  src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
e656c7ec7349b54b07f8c734266445d1fc15fabb 
  src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
1a45d08c8854a335161476c8321c751f763dc12e 
  src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
89140223f4f3acd02ade6fb95734744ef19d89bc 
  
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
 PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
6d2ac494030018b100c1d01ce373c4896f2ed7d2 
  src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
  src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
78a9670573261b2bdae0810c104ca7c1e4736ea8 
  src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 
cbb711fa555c94626963b4d9540a71e6a737abd8 
  src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java 
855573efea68ba4f29b3c7fbfb8da56bbdad352b 
  src/test/java/org/apache/aurora/scheduler/DriverFactoryImplTest.java 
0f6192269cc122580869ae7717acad161fac6f1d 
  src/test/java/org/apache/aurora/scheduler/DriverTest.java 
a96dd87421c32e4da4da4c953dda4c6be4e8e592 
  src/test/java/org/apache/aurora/scheduler/MesosSchedulerImplTest.java 
8dd908e69c71e96d03806e0c4ceb00dab81d756f 
  src/test/java/org/apache/aurora/scheduler/MesosTaskFactoryImplTest.java 
aa026cf3b6ace5e0aaf9b239aff0c2f3210a0883 
  src/test/java/org/apache/aurora/scheduler/SchedulerLifecycleTest.java 
986160133a3471fc36cad2615c2aa1a7d8983b5a 
  src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 
713797182e22956640eeffb796e0a0ff7ec3ddc9 
  

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Aurora ReviewBot

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60402
---

Ship it!


Master (03cb0d1) is green with this patch.
  ./build-support/jenkins/build.sh

- Aurora ReviewBot


On Nov. 7, 2014, 10:03 p.m., Bill Farner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27746/
 ---
 
 (Updated Nov. 7, 2014, 10:03 p.m.)
 
 
 Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
 
 
 Bugs: AURORA-920
 https://issues.apache.org/jira/browse/AURORA-920
 
 
 Repository: aurora
 
 
 Description
 ---
 
 Apologies for the large diff - this is a thread that began with fixing a 
 SchedulerDriver-related race that would exist if we moved to an async event 
 bus.  Most of the green code is not new, but extracted from broken-apart 
 and/or deleted files.  However, i think this refactor makes the lifecycle of 
 the scheduler (and driver) easier to reason about.
 
 Changes of note:
 - We now use AbstractIdleService to manage the call order on the 
 SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
 - Many mesos-specific classes were moved to 
 `org.apache.aurora.scheduler.mesos`
 - Command line args used directly from `DriverFactoryImpl` are now properly 
 injected via `DriverSettings`
 - Untestable code around creating the driver is cordoned off to 
 `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
 constructor
 - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
 and `CommandLineDriverSettingsModule`
 
 
 Diffs
 -
 
   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
   src/main/java/org/apache/aurora/scheduler/Driver.java 
 b07378f3772136177c81a65497152e9d6524b0a7 
   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
 c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
 ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
 453f22a8e97dbf535dd99598332e71e944ebf8d2 
   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
 8d306409bc382fbe5c53bb394d03c923b8f75889 
   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
 048740953629a950c136179c9b880b8dce8fa932 
   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
 feb70a9351fe804b6745304e4cbbf6d010c56579 
   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
 0b023a25292907b8211274f1533d89de7c4b7ce4 
   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
 e656c7ec7349b54b07f8c734266445d1fc15fabb 
   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
 1a45d08c8854a335161476c8321c751f763dc12e 
   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
 89140223f4f3acd02ade6fb95734744ef19d89bc 
   
 src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
  PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
 6d2ac494030018b100c1d01ce373c4896f2ed7d2 
   src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
 fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
 78a9670573261b2bdae0810c104ca7c1e4736ea8 
   src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 
 cbb711fa555c94626963b4d9540a71e6a737abd8 
   
 src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java 
 855573efea68ba4f29b3c7fbfb8da56bbdad352b 
   src/test/java/org/apache/aurora/scheduler/DriverFactoryImplTest.java 
 0f6192269cc122580869ae7717acad161fac6f1d 
   src/test/java/org/apache/aurora/scheduler/DriverTest.java 

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Maxim Khutornenko

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60398
---



src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java
https://reviews.apache.org/r/27746/#comment101785

LOG.info(Retrying for ... in ...)



src/main/java/org/apache/aurora/scheduler/mesos/Driver.java
https://reviews.apache.org/r/27746/#comment101776

is this really spelled as blocK with capital K?



src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java
https://reviews.apache.org/r/27746/#comment101779

Does it have to be public?



src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java
https://reviews.apache.org/r/27746/#comment101788

in(Singleton.class)?



src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java
https://reviews.apache.org/r/27746/#comment101793

Can you elaborate on the rm -rf effect? It's not clear how that happens.


- Maxim Khutornenko


On Nov. 7, 2014, 10:03 p.m., Bill Farner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27746/
 ---
 
 (Updated Nov. 7, 2014, 10:03 p.m.)
 
 
 Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
 
 
 Bugs: AURORA-920
 https://issues.apache.org/jira/browse/AURORA-920
 
 
 Repository: aurora
 
 
 Description
 ---
 
 Apologies for the large diff - this is a thread that began with fixing a 
 SchedulerDriver-related race that would exist if we moved to an async event 
 bus.  Most of the green code is not new, but extracted from broken-apart 
 and/or deleted files.  However, i think this refactor makes the lifecycle of 
 the scheduler (and driver) easier to reason about.
 
 Changes of note:
 - We now use AbstractIdleService to manage the call order on the 
 SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
 - Many mesos-specific classes were moved to 
 `org.apache.aurora.scheduler.mesos`
 - Command line args used directly from `DriverFactoryImpl` are now properly 
 injected via `DriverSettings`
 - Untestable code around creating the driver is cordoned off to 
 `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
 constructor
 - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
 and `CommandLineDriverSettingsModule`
 
 
 Diffs
 -
 
   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
   src/main/java/org/apache/aurora/scheduler/Driver.java 
 b07378f3772136177c81a65497152e9d6524b0a7 
   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
 c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
 ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
 453f22a8e97dbf535dd99598332e71e944ebf8d2 
   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
 8d306409bc382fbe5c53bb394d03c923b8f75889 
   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
 048740953629a950c136179c9b880b8dce8fa932 
   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
 feb70a9351fe804b6745304e4cbbf6d010c56579 
   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
 0b023a25292907b8211274f1533d89de7c4b7ce4 
   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
 e656c7ec7349b54b07f8c734266445d1fc15fabb 
   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
 1a45d08c8854a335161476c8321c751f763dc12e 
   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
 89140223f4f3acd02ade6fb95734744ef19d89bc 
   
 src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
  PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
 PRE-CREATION 
   

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Kevin Sweeney

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60400
---

Ship it!



src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java
https://reviews.apache.org/r/27746/#comment101782

typo in blockUntilStopped



src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java
https://reviews.apache.org/r/27746/#comment101794

Technically it's dynamically-loaded.



src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java
https://reviews.apache.org/r/27746/#comment101796

Futures.getUnchecked



src/test/java/org/apache/aurora/scheduler/app/local/FakeMaster.java
https://reviews.apache.org/r/27746/#comment101802

Futures.getUnchecked


- Kevin Sweeney


On Nov. 7, 2014, 2:03 p.m., Bill Farner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27746/
 ---
 
 (Updated Nov. 7, 2014, 2:03 p.m.)
 
 
 Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
 
 
 Bugs: AURORA-920
 https://issues.apache.org/jira/browse/AURORA-920
 
 
 Repository: aurora
 
 
 Description
 ---
 
 Apologies for the large diff - this is a thread that began with fixing a 
 SchedulerDriver-related race that would exist if we moved to an async event 
 bus.  Most of the green code is not new, but extracted from broken-apart 
 and/or deleted files.  However, i think this refactor makes the lifecycle of 
 the scheduler (and driver) easier to reason about.
 
 Changes of note:
 - We now use AbstractIdleService to manage the call order on the 
 SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
 - Many mesos-specific classes were moved to 
 `org.apache.aurora.scheduler.mesos`
 - Command line args used directly from `DriverFactoryImpl` are now properly 
 injected via `DriverSettings`
 - Untestable code around creating the driver is cordoned off to 
 `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
 constructor
 - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
 and `CommandLineDriverSettingsModule`
 
 
 Diffs
 -
 
   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
   src/main/java/org/apache/aurora/scheduler/Driver.java 
 b07378f3772136177c81a65497152e9d6524b0a7 
   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
 c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
 ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
 453f22a8e97dbf535dd99598332e71e944ebf8d2 
   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
 8d306409bc382fbe5c53bb394d03c923b8f75889 
   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
 048740953629a950c136179c9b880b8dce8fa932 
   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
 feb70a9351fe804b6745304e4cbbf6d010c56579 
   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
 0b023a25292907b8211274f1533d89de7c4b7ce4 
   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
 e656c7ec7349b54b07f8c734266445d1fc15fabb 
   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
 1a45d08c8854a335161476c8321c751f763dc12e 
   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
 89140223f4f3acd02ade6fb95734744ef19d89bc 
   
 src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
  PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
 6d2ac494030018b100c1d01ce373c4896f2ed7d2 
   src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
 fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Bill Farner


 On Nov. 7, 2014, 10:39 p.m., Maxim Khutornenko wrote:
  src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java, line 123
  https://reviews.apache.org/r/27746/diff/1/?file=754888#file754888line123
 
  LOG.info(Retrying for ... in ...)

Done.


 On Nov. 7, 2014, 10:39 p.m., Maxim Khutornenko wrote:
  src/main/java/org/apache/aurora/scheduler/mesos/Driver.java, line 56
  https://reviews.apache.org/r/27746/diff/1/?file=754890#file754890line56
 
  is this really spelled as blocK with capital K?

It is, but it shouldn't be.  Fixed.


 On Nov. 7, 2014, 10:39 p.m., Maxim Khutornenko wrote:
  src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java, line 25
  https://reviews.apache.org/r/27746/diff/1/?file=754893#file754893line25
 
  Does it have to be public?

Yes, it's used by integration tests.  Marked `@VisibleForTesting`.


 On Nov. 7, 2014, 10:39 p.m., Maxim Khutornenko wrote:
  src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java, 
  line 24
  https://reviews.apache.org/r/27746/diff/1/?file=754894#file754894line24
 
  in(Singleton.class)?

Unnecessary, i prefer to avoid singleton binding unless it's warranted.


 On Nov. 7, 2014, 10:39 p.m., Maxim Khutornenko wrote:
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java,
   line 111
  https://reviews.apache.org/r/27746/diff/1/?file=754896#file754896line111
 
  Can you elaborate on the rm -rf effect? It's not clear how that 
  happens.

I'll refer to the mesos docs at risk of repeating them.


- Bill


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60398
---


On Nov. 7, 2014, 10:03 p.m., Bill Farner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27746/
 ---
 
 (Updated Nov. 7, 2014, 10:03 p.m.)
 
 
 Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
 
 
 Bugs: AURORA-920
 https://issues.apache.org/jira/browse/AURORA-920
 
 
 Repository: aurora
 
 
 Description
 ---
 
 Apologies for the large diff - this is a thread that began with fixing a 
 SchedulerDriver-related race that would exist if we moved to an async event 
 bus.  Most of the green code is not new, but extracted from broken-apart 
 and/or deleted files.  However, i think this refactor makes the lifecycle of 
 the scheduler (and driver) easier to reason about.
 
 Changes of note:
 - We now use AbstractIdleService to manage the call order on the 
 SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
 - Many mesos-specific classes were moved to 
 `org.apache.aurora.scheduler.mesos`
 - Command line args used directly from `DriverFactoryImpl` are now properly 
 injected via `DriverSettings`
 - Untestable code around creating the driver is cordoned off to 
 `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
 constructor
 - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
 and `CommandLineDriverSettingsModule`
 
 
 Diffs
 -
 
   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
   src/main/java/org/apache/aurora/scheduler/Driver.java 
 b07378f3772136177c81a65497152e9d6524b0a7 
   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
 c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
 ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
 453f22a8e97dbf535dd99598332e71e944ebf8d2 
   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
 8d306409bc382fbe5c53bb394d03c923b8f75889 
   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
 048740953629a950c136179c9b880b8dce8fa932 
   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
 feb70a9351fe804b6745304e4cbbf6d010c56579 
   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
 0b023a25292907b8211274f1533d89de7c4b7ce4 
   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
 e656c7ec7349b54b07f8c734266445d1fc15fabb 
   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
 1a45d08c8854a335161476c8321c751f763dc12e 
   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
 89140223f4f3acd02ade6fb95734744ef19d89bc 
   
 

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Bill Farner


 On Nov. 7, 2014, 10:47 p.m., Kevin Sweeney wrote:
  src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java, line 326
  https://reviews.apache.org/r/27746/diff/1/?file=754879#file754879line326
 
  typo in blockUntilStopped

Fixed.


 On Nov. 7, 2014, 10:47 p.m., Kevin Sweeney wrote:
  src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java, 
  line 19
  https://reviews.apache.org/r/27746/diff/1/?file=754894#file754894line19
 
  Technically it's dynamically-loaded.

Better yet, it's statically dynamically loaded :-)
static {
  MesosNativeLibrary.load();
}

I'll avoid the wording nuance.


- Bill


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60400
---


On Nov. 7, 2014, 10:03 p.m., Bill Farner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27746/
 ---
 
 (Updated Nov. 7, 2014, 10:03 p.m.)
 
 
 Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
 
 
 Bugs: AURORA-920
 https://issues.apache.org/jira/browse/AURORA-920
 
 
 Repository: aurora
 
 
 Description
 ---
 
 Apologies for the large diff - this is a thread that began with fixing a 
 SchedulerDriver-related race that would exist if we moved to an async event 
 bus.  Most of the green code is not new, but extracted from broken-apart 
 and/or deleted files.  However, i think this refactor makes the lifecycle of 
 the scheduler (and driver) easier to reason about.
 
 Changes of note:
 - We now use AbstractIdleService to manage the call order on the 
 SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
 - Many mesos-specific classes were moved to 
 `org.apache.aurora.scheduler.mesos`
 - Command line args used directly from `DriverFactoryImpl` are now properly 
 injected via `DriverSettings`
 - Untestable code around creating the driver is cordoned off to 
 `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
 constructor
 - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
 and `CommandLineDriverSettingsModule`
 
 
 Diffs
 -
 
   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
   src/main/java/org/apache/aurora/scheduler/Driver.java 
 b07378f3772136177c81a65497152e9d6524b0a7 
   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
 c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
 ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
 453f22a8e97dbf535dd99598332e71e944ebf8d2 
   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
 8d306409bc382fbe5c53bb394d03c923b8f75889 
   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
 048740953629a950c136179c9b880b8dce8fa932 
   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
 feb70a9351fe804b6745304e4cbbf6d010c56579 
   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
 0b023a25292907b8211274f1533d89de7c4b7ce4 
   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
 e656c7ec7349b54b07f8c734266445d1fc15fabb 
   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
 1a45d08c8854a335161476c8321c751f763dc12e 
   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
 89140223f4f3acd02ade6fb95734744ef19d89bc 
   
 src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
  PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
 6d2ac494030018b100c1d01ce373c4896f2ed7d2 
   src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
 

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Bill Farner

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/
---

(Updated Nov. 7, 2014, 11:05 p.m.)


Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.


Bugs: AURORA-920
https://issues.apache.org/jira/browse/AURORA-920


Repository: aurora


Description
---

Apologies for the large diff - this is a thread that began with fixing a 
SchedulerDriver-related race that would exist if we moved to an async event 
bus.  Most of the green code is not new, but extracted from broken-apart and/or 
deleted files.  However, i think this refactor makes the lifecycle of the 
scheduler (and driver) easier to reason about.

Changes of note:
- We now use AbstractIdleService to manage the call order on the 
SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
- Many mesos-specific classes were moved to `org.apache.aurora.scheduler.mesos`
- Command line args used directly from `DriverFactoryImpl` are now properly 
injected via `DriverSettings`
- Untestable code around creating the driver is cordoned off to 
`DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
constructor
- Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
and `CommandLineDriverSettingsModule`


Diffs (updated)
-

  config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
  src/main/java/org/apache/aurora/scheduler/Driver.java 
b07378f3772136177c81a65497152e9d6524b0a7 
  src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
9cc04a84a37374ffca418e2ff767992ee23b9f3e 
  src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
  src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
  src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
453f22a8e97dbf535dd99598332e71e944ebf8d2 
  src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
8d306409bc382fbe5c53bb394d03c923b8f75889 
  src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
9ec3f412a9a9210279a2ce666bd99f61481ab92d 
  src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
048740953629a950c136179c9b880b8dce8fa932 
  src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
  src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
feb70a9351fe804b6745304e4cbbf6d010c56579 
  src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
0b023a25292907b8211274f1533d89de7c4b7ce4 
  src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
e656c7ec7349b54b07f8c734266445d1fc15fabb 
  src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
1a45d08c8854a335161476c8321c751f763dc12e 
  src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
89140223f4f3acd02ade6fb95734744ef19d89bc 
  
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
 PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
6d2ac494030018b100c1d01ce373c4896f2ed7d2 
  src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
  src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
78a9670573261b2bdae0810c104ca7c1e4736ea8 
  src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 
cbb711fa555c94626963b4d9540a71e6a737abd8 
  src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java 
855573efea68ba4f29b3c7fbfb8da56bbdad352b 
  src/test/java/org/apache/aurora/scheduler/DriverFactoryImplTest.java 
0f6192269cc122580869ae7717acad161fac6f1d 
  src/test/java/org/apache/aurora/scheduler/DriverTest.java 
a96dd87421c32e4da4da4c953dda4c6be4e8e592 
  src/test/java/org/apache/aurora/scheduler/MesosSchedulerImplTest.java 
8dd908e69c71e96d03806e0c4ceb00dab81d756f 
  src/test/java/org/apache/aurora/scheduler/MesosTaskFactoryImplTest.java 
aa026cf3b6ace5e0aaf9b239aff0c2f3210a0883 
  src/test/java/org/apache/aurora/scheduler/SchedulerLifecycleTest.java 
986160133a3471fc36cad2615c2aa1a7d8983b5a 
  src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Aurora ReviewBot

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60413
---


Master (03cb0d1) is red with this patch.
  ./build-support/jenkins/build.sh

  File /usr/lib/python2.7/sysconfig.py, line 355, in _init_posix
raise IOError(msg)
IOError: invalid Python installation: unable to open 
/x1/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/local/include/python2.7/pyconfig.h
 (No such file or directory)

Build operating on top level addresses: 
set([BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/BUILD,
 all)])
Problem executing PythonBuilder for targets 
OrderedSet([PythonTests(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/admin/BUILD,
 host_maintenance)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/BUILD,
 api)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 api)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 restarter)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 instance_watcher)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 scheduler_client)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/au
 th/BUILD, auth)), 
PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD,
 py-thrift)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 cluster)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 transport)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 scheduler_mux)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 error_handling_thread)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 task_util)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/BUILD,
 base)), PythonLibrary
 
(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 http_signaler)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 updater_util)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 sla)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 common)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 aurora_job_key)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 cluster_option)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 clusters)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/
 workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, shellify)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD,
 py-thrift-packaged)), 
PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD,
 py-thrift-test)), 
PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD,
 py-thrift-storage)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 updater)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 job_monitor)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 quota_check)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace
 /AuroraBot/src/main/python/apache/aurora/admin/BUILD, host_maintenance)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/admin/BUILD,
 util)), 

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Bill Farner

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/
---

(Updated Nov. 7, 2014, 10:59 p.m.)


Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.


Bugs: AURORA-920
https://issues.apache.org/jira/browse/AURORA-920


Repository: aurora


Description
---

Apologies for the large diff - this is a thread that began with fixing a 
SchedulerDriver-related race that would exist if we moved to an async event 
bus.  Most of the green code is not new, but extracted from broken-apart and/or 
deleted files.  However, i think this refactor makes the lifecycle of the 
scheduler (and driver) easier to reason about.

Changes of note:
- We now use AbstractIdleService to manage the call order on the 
SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
- Many mesos-specific classes were moved to `org.apache.aurora.scheduler.mesos`
- Command line args used directly from `DriverFactoryImpl` are now properly 
injected via `DriverSettings`
- Untestable code around creating the driver is cordoned off to 
`DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
constructor
- Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
and `CommandLineDriverSettingsModule`


Diffs (updated)
-

  config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
  src/main/java/org/apache/aurora/scheduler/Driver.java 
b07378f3772136177c81a65497152e9d6524b0a7 
  src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
9cc04a84a37374ffca418e2ff767992ee23b9f3e 
  src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
  src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
  src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
453f22a8e97dbf535dd99598332e71e944ebf8d2 
  src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
8d306409bc382fbe5c53bb394d03c923b8f75889 
  src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
9ec3f412a9a9210279a2ce666bd99f61481ab92d 
  src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
048740953629a950c136179c9b880b8dce8fa932 
  src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
  src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
feb70a9351fe804b6745304e4cbbf6d010c56579 
  src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
0b023a25292907b8211274f1533d89de7c4b7ce4 
  src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
e656c7ec7349b54b07f8c734266445d1fc15fabb 
  src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
1a45d08c8854a335161476c8321c751f763dc12e 
  src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
89140223f4f3acd02ade6fb95734744ef19d89bc 
  
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
 PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
6d2ac494030018b100c1d01ce373c4896f2ed7d2 
  src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
  src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
78a9670573261b2bdae0810c104ca7c1e4736ea8 
  src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 
cbb711fa555c94626963b4d9540a71e6a737abd8 
  src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java 
855573efea68ba4f29b3c7fbfb8da56bbdad352b 
  src/test/java/org/apache/aurora/scheduler/DriverFactoryImplTest.java 
0f6192269cc122580869ae7717acad161fac6f1d 
  src/test/java/org/apache/aurora/scheduler/DriverTest.java 
a96dd87421c32e4da4da4c953dda4c6be4e8e592 
  src/test/java/org/apache/aurora/scheduler/MesosSchedulerImplTest.java 
8dd908e69c71e96d03806e0c4ceb00dab81d756f 
  src/test/java/org/apache/aurora/scheduler/MesosTaskFactoryImplTest.java 
aa026cf3b6ace5e0aaf9b239aff0c2f3210a0883 
  src/test/java/org/apache/aurora/scheduler/SchedulerLifecycleTest.java 
986160133a3471fc36cad2615c2aa1a7d8983b5a 
  src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Maxim Khutornenko

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60414
---

Ship it!


Ship It!

- Maxim Khutornenko


On Nov. 7, 2014, 11:05 p.m., Bill Farner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27746/
 ---
 
 (Updated Nov. 7, 2014, 11:05 p.m.)
 
 
 Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
 
 
 Bugs: AURORA-920
 https://issues.apache.org/jira/browse/AURORA-920
 
 
 Repository: aurora
 
 
 Description
 ---
 
 Apologies for the large diff - this is a thread that began with fixing a 
 SchedulerDriver-related race that would exist if we moved to an async event 
 bus.  Most of the green code is not new, but extracted from broken-apart 
 and/or deleted files.  However, i think this refactor makes the lifecycle of 
 the scheduler (and driver) easier to reason about.
 
 Changes of note:
 - We now use AbstractIdleService to manage the call order on the 
 SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
 - Many mesos-specific classes were moved to 
 `org.apache.aurora.scheduler.mesos`
 - Command line args used directly from `DriverFactoryImpl` are now properly 
 injected via `DriverSettings`
 - Untestable code around creating the driver is cordoned off to 
 `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
 constructor
 - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
 and `CommandLineDriverSettingsModule`
 
 
 Diffs
 -
 
   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
   src/main/java/org/apache/aurora/scheduler/Driver.java 
 b07378f3772136177c81a65497152e9d6524b0a7 
   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
 c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
 ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
 453f22a8e97dbf535dd99598332e71e944ebf8d2 
   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
 8d306409bc382fbe5c53bb394d03c923b8f75889 
   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
 048740953629a950c136179c9b880b8dce8fa932 
   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
 feb70a9351fe804b6745304e4cbbf6d010c56579 
   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
 0b023a25292907b8211274f1533d89de7c4b7ce4 
   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
 e656c7ec7349b54b07f8c734266445d1fc15fabb 
   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
 1a45d08c8854a335161476c8321c751f763dc12e 
   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
 89140223f4f3acd02ade6fb95734744ef19d89bc 
   
 src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
  PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
 6d2ac494030018b100c1d01ce373c4896f2ed7d2 
   src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
 fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
 78a9670573261b2bdae0810c104ca7c1e4736ea8 
   src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 
 cbb711fa555c94626963b4d9540a71e6a737abd8 
   
 src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java 
 855573efea68ba4f29b3c7fbfb8da56bbdad352b 
   src/test/java/org/apache/aurora/scheduler/DriverFactoryImplTest.java 
 0f6192269cc122580869ae7717acad161fac6f1d 
   src/test/java/org/apache/aurora/scheduler/DriverTest.java 
 a96dd87421c32e4da4da4c953dda4c6be4e8e592 
   

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Kevin Sweeney


 On Nov. 7, 2014, 2:39 p.m., Maxim Khutornenko wrote:
  src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java, line 123
  https://reviews.apache.org/r/27746/diff/1/?file=754888#file754888line123
 
  LOG.info(Retrying for ... in ...)
 
 Bill Farner wrote:
 Done.

Not worth such a high log level IMO - maybe fine. We expect this to happen 
frequently if startup takes longer than the initial delay.


- Kevin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60398
---


On Nov. 7, 2014, 2:59 p.m., Bill Farner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27746/
 ---
 
 (Updated Nov. 7, 2014, 2:59 p.m.)
 
 
 Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
 
 
 Bugs: AURORA-920
 https://issues.apache.org/jira/browse/AURORA-920
 
 
 Repository: aurora
 
 
 Description
 ---
 
 Apologies for the large diff - this is a thread that began with fixing a 
 SchedulerDriver-related race that would exist if we moved to an async event 
 bus.  Most of the green code is not new, but extracted from broken-apart 
 and/or deleted files.  However, i think this refactor makes the lifecycle of 
 the scheduler (and driver) easier to reason about.
 
 Changes of note:
 - We now use AbstractIdleService to manage the call order on the 
 SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
 - Many mesos-specific classes were moved to 
 `org.apache.aurora.scheduler.mesos`
 - Command line args used directly from `DriverFactoryImpl` are now properly 
 injected via `DriverSettings`
 - Untestable code around creating the driver is cordoned off to 
 `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
 constructor
 - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
 and `CommandLineDriverSettingsModule`
 
 
 Diffs
 -
 
   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
   src/main/java/org/apache/aurora/scheduler/Driver.java 
 b07378f3772136177c81a65497152e9d6524b0a7 
   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
 c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
 ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
 453f22a8e97dbf535dd99598332e71e944ebf8d2 
   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
 8d306409bc382fbe5c53bb394d03c923b8f75889 
   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
 048740953629a950c136179c9b880b8dce8fa932 
   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
 feb70a9351fe804b6745304e4cbbf6d010c56579 
   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
 0b023a25292907b8211274f1533d89de7c4b7ce4 
   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
 e656c7ec7349b54b07f8c734266445d1fc15fabb 
   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
 1a45d08c8854a335161476c8321c751f763dc12e 
   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
 89140223f4f3acd02ade6fb95734744ef19d89bc 
   
 src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
  PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
 6d2ac494030018b100c1d01ce373c4896f2ed7d2 
   src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
 fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
 78a9670573261b2bdae0810c104ca7c1e4736ea8 
   src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 
 

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Bill Farner


 On Nov. 7, 2014, 10:39 p.m., Maxim Khutornenko wrote:
  src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java, line 123
  https://reviews.apache.org/r/27746/diff/1/?file=754888#file754888line123
 
  LOG.info(Retrying for ... in ...)
 
 Bill Farner wrote:
 Done.
 
 Kevin Sweeney wrote:
 Not worth such a high log level IMO - maybe fine. We expect this to 
 happen frequently if startup takes longer than the initial delay.

Sure, i'll move it down to fine.


- Bill


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60398
---


On Nov. 7, 2014, 10:59 p.m., Bill Farner wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/27746/
 ---
 
 (Updated Nov. 7, 2014, 10:59 p.m.)
 
 
 Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
 
 
 Bugs: AURORA-920
 https://issues.apache.org/jira/browse/AURORA-920
 
 
 Repository: aurora
 
 
 Description
 ---
 
 Apologies for the large diff - this is a thread that began with fixing a 
 SchedulerDriver-related race that would exist if we moved to an async event 
 bus.  Most of the green code is not new, but extracted from broken-apart 
 and/or deleted files.  However, i think this refactor makes the lifecycle of 
 the scheduler (and driver) easier to reason about.
 
 Changes of note:
 - We now use AbstractIdleService to manage the call order on the 
 SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
 - Many mesos-specific classes were moved to 
 `org.apache.aurora.scheduler.mesos`
 - Command line args used directly from `DriverFactoryImpl` are now properly 
 injected via `DriverSettings`
 - Untestable code around creating the driver is cordoned off to 
 `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
 constructor
 - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
 and `CommandLineDriverSettingsModule`
 
 
 Diffs
 -
 
   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
   src/main/java/org/apache/aurora/scheduler/Driver.java 
 b07378f3772136177c81a65497152e9d6524b0a7 
   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
 c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
 ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
 453f22a8e97dbf535dd99598332e71e944ebf8d2 
   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
 8d306409bc382fbe5c53bb394d03c923b8f75889 
   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
 048740953629a950c136179c9b880b8dce8fa932 
   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
 feb70a9351fe804b6745304e4cbbf6d010c56579 
   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
 0b023a25292907b8211274f1533d89de7c4b7ce4 
   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
 e656c7ec7349b54b07f8c734266445d1fc15fabb 
   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
 1a45d08c8854a335161476c8321c751f763dc12e 
   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
 89140223f4f3acd02ade6fb95734744ef19d89bc 
   
 src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
  PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
 PRE-CREATION 
   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
 6d2ac494030018b100c1d01ce373c4896f2ed7d2 
   src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
 fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
 78a9670573261b2bdae0810c104ca7c1e4736ea8