Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Bill Farner 


> On Nov. 7, 2014, 10:39 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java, line 123
> > 
> >
> > LOG.info("Retrying for ... in ...")
> 
> Bill Farner wrote:
> Done.
> 
> Kevin Sweeney wrote:
> Not worth such a high log level IMO - maybe fine. We expect this to 
> happen frequently if startup takes longer than the initial delay.

Sure, i'll move it down to fine.


- Bill


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60398
---


On Nov. 7, 2014, 10:59 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27746/
> ---
> 
> (Updated Nov. 7, 2014, 10:59 p.m.)
> 
> 
> Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-920
> https://issues.apache.org/jira/browse/AURORA-920
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Apologies for the large diff - this is a thread that began with fixing a 
> SchedulerDriver-related race that would exist if we moved to an async event 
> bus.  Most of the green code is not new, but extracted from broken-apart 
> and/or deleted files.  However, i think this refactor makes the lifecycle of 
> the scheduler (and driver) easier to reason about.
> 
> Changes of note:
> - We now use AbstractIdleService to manage the call order on the 
> SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
> - Many mesos-specific classes were moved to 
> `org.apache.aurora.scheduler.mesos`
> - Command line args used directly from `DriverFactoryImpl` are now properly 
> injected via `DriverSettings`
> - Untestable code around creating the driver is cordoned off to 
> `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
> constructor
> - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
> and `CommandLineDriverSettingsModule`
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
>   src/main/java/org/apache/aurora/scheduler/Driver.java 
> b07378f3772136177c81a65497152e9d6524b0a7 
>   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
> 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
>   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
> c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
>   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
> ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
>   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
> 453f22a8e97dbf535dd99598332e71e944ebf8d2 
>   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
> 8d306409bc382fbe5c53bb394d03c923b8f75889 
>   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
> 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
>   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
> 048740953629a950c136179c9b880b8dce8fa932 
>   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
> 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> feb70a9351fe804b6745304e4cbbf6d010c56579 
>   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
> 0b023a25292907b8211274f1533d89de7c4b7ce4 
>   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
> e656c7ec7349b54b07f8c734266445d1fc15fabb 
>   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
> 1a45d08c8854a335161476c8321c751f763dc12e 
>   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
> 89140223f4f3acd02ade6fb95734744ef19d89bc 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
> 6d2ac494030018b100c1d01ce373c4896f2ed7d2 
>   src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
> fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
>   src/main/java/org/apache/aurora/sched

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Kevin Sweeney 


> On Nov. 7, 2014, 2:39 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java, line 123
> > 
> >
> > LOG.info("Retrying for ... in ...")
> 
> Bill Farner wrote:
> Done.

Not worth such a high log level IMO - maybe fine. We expect this to happen 
frequently if startup takes longer than the initial delay.


- Kevin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60398
---


On Nov. 7, 2014, 2:59 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27746/
> ---
> 
> (Updated Nov. 7, 2014, 2:59 p.m.)
> 
> 
> Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-920
> https://issues.apache.org/jira/browse/AURORA-920
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Apologies for the large diff - this is a thread that began with fixing a 
> SchedulerDriver-related race that would exist if we moved to an async event 
> bus.  Most of the green code is not new, but extracted from broken-apart 
> and/or deleted files.  However, i think this refactor makes the lifecycle of 
> the scheduler (and driver) easier to reason about.
> 
> Changes of note:
> - We now use AbstractIdleService to manage the call order on the 
> SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
> - Many mesos-specific classes were moved to 
> `org.apache.aurora.scheduler.mesos`
> - Command line args used directly from `DriverFactoryImpl` are now properly 
> injected via `DriverSettings`
> - Untestable code around creating the driver is cordoned off to 
> `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
> constructor
> - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
> and `CommandLineDriverSettingsModule`
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
>   src/main/java/org/apache/aurora/scheduler/Driver.java 
> b07378f3772136177c81a65497152e9d6524b0a7 
>   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
> 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
>   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
> c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
>   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
> ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
>   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
> 453f22a8e97dbf535dd99598332e71e944ebf8d2 
>   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
> 8d306409bc382fbe5c53bb394d03c923b8f75889 
>   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
> 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
>   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
> 048740953629a950c136179c9b880b8dce8fa932 
>   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
> 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> feb70a9351fe804b6745304e4cbbf6d010c56579 
>   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
> 0b023a25292907b8211274f1533d89de7c4b7ce4 
>   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
> e656c7ec7349b54b07f8c734266445d1fc15fabb 
>   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
> 1a45d08c8854a335161476c8321c751f763dc12e 
>   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
> 89140223f4f3acd02ade6fb95734744ef19d89bc 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
> 6d2ac494030018b100c1d01ce373c4896f2ed7d2 
>   src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
> fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
>   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
> 78a9670573261b2bdae0810c104ca7c1e4736

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Maxim Khutornenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60414
---

Ship it!


Ship It!

- Maxim Khutornenko


On Nov. 7, 2014, 11:05 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27746/
> ---
> 
> (Updated Nov. 7, 2014, 11:05 p.m.)
> 
> 
> Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-920
> https://issues.apache.org/jira/browse/AURORA-920
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Apologies for the large diff - this is a thread that began with fixing a 
> SchedulerDriver-related race that would exist if we moved to an async event 
> bus.  Most of the green code is not new, but extracted from broken-apart 
> and/or deleted files.  However, i think this refactor makes the lifecycle of 
> the scheduler (and driver) easier to reason about.
> 
> Changes of note:
> - We now use AbstractIdleService to manage the call order on the 
> SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
> - Many mesos-specific classes were moved to 
> `org.apache.aurora.scheduler.mesos`
> - Command line args used directly from `DriverFactoryImpl` are now properly 
> injected via `DriverSettings`
> - Untestable code around creating the driver is cordoned off to 
> `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
> constructor
> - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
> and `CommandLineDriverSettingsModule`
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
>   src/main/java/org/apache/aurora/scheduler/Driver.java 
> b07378f3772136177c81a65497152e9d6524b0a7 
>   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
> 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
>   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
> c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
>   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
> ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
>   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
> 453f22a8e97dbf535dd99598332e71e944ebf8d2 
>   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
> 8d306409bc382fbe5c53bb394d03c923b8f75889 
>   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
> 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
>   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
> 048740953629a950c136179c9b880b8dce8fa932 
>   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
> 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> feb70a9351fe804b6745304e4cbbf6d010c56579 
>   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
> 0b023a25292907b8211274f1533d89de7c4b7ce4 
>   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
> e656c7ec7349b54b07f8c734266445d1fc15fabb 
>   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
> 1a45d08c8854a335161476c8321c751f763dc12e 
>   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
> 89140223f4f3acd02ade6fb95734744ef19d89bc 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
> 6d2ac494030018b100c1d01ce373c4896f2ed7d2 
>   src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
> fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
>   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
> 78a9670573261b2bdae0810c104ca7c1e4736ea8 
>   src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 
> cbb711fa555c94626963b4d9540a71e6a737abd8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java 
> 855573efea68ba4f29b3c7fbfb8da56bbdad352b 
>   src/test/java/org/apache/aurora/scheduler/DriverFactoryImplTest.java 
> 0f6192269cc122580869ae7717acad161fac6f1d 
>   src/test/java/org/apac

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Bill Farner 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/
---

(Updated Nov. 7, 2014, 10:59 p.m.)


Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.


Bugs: AURORA-920
https://issues.apache.org/jira/browse/AURORA-920


Repository: aurora


Description
---

Apologies for the large diff - this is a thread that began with fixing a 
SchedulerDriver-related race that would exist if we moved to an async event 
bus.  Most of the green code is not new, but extracted from broken-apart and/or 
deleted files.  However, i think this refactor makes the lifecycle of the 
scheduler (and driver) easier to reason about.

Changes of note:
- We now use AbstractIdleService to manage the call order on the 
SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
- Many mesos-specific classes were moved to `org.apache.aurora.scheduler.mesos`
- Command line args used directly from `DriverFactoryImpl` are now properly 
injected via `DriverSettings`
- Untestable code around creating the driver is cordoned off to 
`DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
constructor
- Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
and `CommandLineDriverSettingsModule`


Diffs (updated)
-

  config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
  src/main/java/org/apache/aurora/scheduler/Driver.java 
b07378f3772136177c81a65497152e9d6524b0a7 
  src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
9cc04a84a37374ffca418e2ff767992ee23b9f3e 
  src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
  src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
  src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
453f22a8e97dbf535dd99598332e71e944ebf8d2 
  src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
8d306409bc382fbe5c53bb394d03c923b8f75889 
  src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
9ec3f412a9a9210279a2ce666bd99f61481ab92d 
  src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
048740953629a950c136179c9b880b8dce8fa932 
  src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
  src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
feb70a9351fe804b6745304e4cbbf6d010c56579 
  src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
0b023a25292907b8211274f1533d89de7c4b7ce4 
  src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
e656c7ec7349b54b07f8c734266445d1fc15fabb 
  src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
1a45d08c8854a335161476c8321c751f763dc12e 
  src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
89140223f4f3acd02ade6fb95734744ef19d89bc 
  
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
 PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
6d2ac494030018b100c1d01ce373c4896f2ed7d2 
  src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
  src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
78a9670573261b2bdae0810c104ca7c1e4736ea8 
  src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 
cbb711fa555c94626963b4d9540a71e6a737abd8 
  src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java 
855573efea68ba4f29b3c7fbfb8da56bbdad352b 
  src/test/java/org/apache/aurora/scheduler/DriverFactoryImplTest.java 
0f6192269cc122580869ae7717acad161fac6f1d 
  src/test/java/org/apache/aurora/scheduler/DriverTest.java 
a96dd87421c32e4da4da4c953dda4c6be4e8e592 
  src/test/java/org/apache/aurora/scheduler/MesosSchedulerImplTest.java 
8dd908e69c71e96d03806e0c4ceb00dab81d756f 
  src/test/java/org/apache/aurora/scheduler/MesosTaskFactoryImplTest.java 
aa026cf3b6ace5e0aaf9b239aff0c2f3210a0883 
  src/test/java/org/apache/aurora/scheduler/SchedulerLifecycleTest.java 
986160133a3471fc36cad2615c2aa1a7d8983b5a 
  src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 
713797182e22956640eeffb796e0a

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60413
---


Master (03cb0d1) is red with this patch.
  ./build-support/jenkins/build.sh

  File "/usr/lib/python2.7/sysconfig.py", line 355, in _init_posix
raise IOError(msg)
IOError: invalid Python installation: unable to open 
/x1/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/local/include/python2.7/pyconfig.h
 (No such file or directory)

Build operating on top level addresses: 
set([BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/BUILD,
 all)])
Problem executing PythonBuilder for targets 
OrderedSet([PythonTests(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/admin/BUILD,
 host_maintenance)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/BUILD,
 api)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 api)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 restarter)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 instance_watcher)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 scheduler_client)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/au
 th/BUILD, auth)), 
PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD,
 py-thrift)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 cluster)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 transport)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 scheduler_mux)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 error_handling_thread)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 task_util)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/BUILD,
 base)), PythonLibrary
 
(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 http_signaler)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 updater_util)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 sla)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 common)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 aurora_job_key)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 cluster_option)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD,
 clusters)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/
 workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, shellify)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD,
 py-thrift-packaged)), 
PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD,
 py-thrift-test)), 
PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD,
 py-thrift-storage)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 updater)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 job_monitor)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD,
 quota_check)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace
 /AuroraBot/src/main/python/apache/aurora/admin/BUILD, host_maintenance)), 
PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/admin/BUILD,
 util)), 
PythonTests(Bui

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Bill Farner 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/
---

(Updated Nov. 7, 2014, 11:05 p.m.)


Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.


Bugs: AURORA-920
https://issues.apache.org/jira/browse/AURORA-920


Repository: aurora


Description
---

Apologies for the large diff - this is a thread that began with fixing a 
SchedulerDriver-related race that would exist if we moved to an async event 
bus.  Most of the green code is not new, but extracted from broken-apart and/or 
deleted files.  However, i think this refactor makes the lifecycle of the 
scheduler (and driver) easier to reason about.

Changes of note:
- We now use AbstractIdleService to manage the call order on the 
SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
- Many mesos-specific classes were moved to `org.apache.aurora.scheduler.mesos`
- Command line args used directly from `DriverFactoryImpl` are now properly 
injected via `DriverSettings`
- Untestable code around creating the driver is cordoned off to 
`DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
constructor
- Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
and `CommandLineDriverSettingsModule`


Diffs (updated)
-

  config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
  src/main/java/org/apache/aurora/scheduler/Driver.java 
b07378f3772136177c81a65497152e9d6524b0a7 
  src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
9cc04a84a37374ffca418e2ff767992ee23b9f3e 
  src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
  src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
  src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
453f22a8e97dbf535dd99598332e71e944ebf8d2 
  src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
8d306409bc382fbe5c53bb394d03c923b8f75889 
  src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
9ec3f412a9a9210279a2ce666bd99f61481ab92d 
  src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
048740953629a950c136179c9b880b8dce8fa932 
  src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
  src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
feb70a9351fe804b6745304e4cbbf6d010c56579 
  src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
0b023a25292907b8211274f1533d89de7c4b7ce4 
  src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
e656c7ec7349b54b07f8c734266445d1fc15fabb 
  src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
1a45d08c8854a335161476c8321c751f763dc12e 
  src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
89140223f4f3acd02ade6fb95734744ef19d89bc 
  
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
 PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
6d2ac494030018b100c1d01ce373c4896f2ed7d2 
  src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
  src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
78a9670573261b2bdae0810c104ca7c1e4736ea8 
  src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 
cbb711fa555c94626963b4d9540a71e6a737abd8 
  src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java 
855573efea68ba4f29b3c7fbfb8da56bbdad352b 
  src/test/java/org/apache/aurora/scheduler/DriverFactoryImplTest.java 
0f6192269cc122580869ae7717acad161fac6f1d 
  src/test/java/org/apache/aurora/scheduler/DriverTest.java 
a96dd87421c32e4da4da4c953dda4c6be4e8e592 
  src/test/java/org/apache/aurora/scheduler/MesosSchedulerImplTest.java 
8dd908e69c71e96d03806e0c4ceb00dab81d756f 
  src/test/java/org/apache/aurora/scheduler/MesosTaskFactoryImplTest.java 
aa026cf3b6ace5e0aaf9b239aff0c2f3210a0883 
  src/test/java/org/apache/aurora/scheduler/SchedulerLifecycleTest.java 
986160133a3471fc36cad2615c2aa1a7d8983b5a 
  src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 
713797182e22956640eeffb796e0a

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Bill Farner 


> On Nov. 7, 2014, 10:47 p.m., Kevin Sweeney wrote:
> > src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java, line 326
> > 
> >
> > typo in blockUntilStopped

Fixed.


> On Nov. 7, 2014, 10:47 p.m., Kevin Sweeney wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java, 
> > line 19
> > 
> >
> > Technically it's dynamically-loaded.

Better yet, it's statically dynamically loaded :-)
static {
  MesosNativeLibrary.load();
}

I'll avoid the wording nuance.


- Bill


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60400
---


On Nov. 7, 2014, 10:03 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27746/
> ---
> 
> (Updated Nov. 7, 2014, 10:03 p.m.)
> 
> 
> Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-920
> https://issues.apache.org/jira/browse/AURORA-920
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Apologies for the large diff - this is a thread that began with fixing a 
> SchedulerDriver-related race that would exist if we moved to an async event 
> bus.  Most of the green code is not new, but extracted from broken-apart 
> and/or deleted files.  However, i think this refactor makes the lifecycle of 
> the scheduler (and driver) easier to reason about.
> 
> Changes of note:
> - We now use AbstractIdleService to manage the call order on the 
> SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
> - Many mesos-specific classes were moved to 
> `org.apache.aurora.scheduler.mesos`
> - Command line args used directly from `DriverFactoryImpl` are now properly 
> injected via `DriverSettings`
> - Untestable code around creating the driver is cordoned off to 
> `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
> constructor
> - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
> and `CommandLineDriverSettingsModule`
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
>   src/main/java/org/apache/aurora/scheduler/Driver.java 
> b07378f3772136177c81a65497152e9d6524b0a7 
>   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
> 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
>   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
> c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
>   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
> ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
>   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
> 453f22a8e97dbf535dd99598332e71e944ebf8d2 
>   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
> 8d306409bc382fbe5c53bb394d03c923b8f75889 
>   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
> 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
>   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
> 048740953629a950c136179c9b880b8dce8fa932 
>   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
> 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> feb70a9351fe804b6745304e4cbbf6d010c56579 
>   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
> 0b023a25292907b8211274f1533d89de7c4b7ce4 
>   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
> e656c7ec7349b54b07f8c734266445d1fc15fabb 
>   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
> 1a45d08c8854a335161476c8321c751f763dc12e 
>   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
> 89140223f4f3acd02ade6fb95734744ef19d89bc 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
> 6d2ac494030018b100c1d01ce373c48

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Bill Farner 


> On Nov. 7, 2014, 10:39 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java, line 123
> > 
> >
> > LOG.info("Retrying for ... in ...")

Done.


> On Nov. 7, 2014, 10:39 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/Driver.java, line 56
> > 
> >
> > is this really spelled as "blocK" with capital K?

It is, but it shouldn't be.  Fixed.


> On Nov. 7, 2014, 10:39 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java, line 25
> > 
> >
> > Does it have to be public?

Yes, it's used by integration tests.  Marked `@VisibleForTesting`.


> On Nov. 7, 2014, 10:39 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java, 
> > line 24
> > 
> >
> > in(Singleton.class)?

Unnecessary, i prefer to avoid singleton binding unless it's warranted.


> On Nov. 7, 2014, 10:39 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java,
> >  line 111
> > 
> >
> > Can you elaborate on the "rm -rf" effect? It's not clear how that 
> > happens.

I'll refer to the mesos docs at risk of repeating them.


- Bill


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60398
---


On Nov. 7, 2014, 10:03 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27746/
> ---
> 
> (Updated Nov. 7, 2014, 10:03 p.m.)
> 
> 
> Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-920
> https://issues.apache.org/jira/browse/AURORA-920
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Apologies for the large diff - this is a thread that began with fixing a 
> SchedulerDriver-related race that would exist if we moved to an async event 
> bus.  Most of the green code is not new, but extracted from broken-apart 
> and/or deleted files.  However, i think this refactor makes the lifecycle of 
> the scheduler (and driver) easier to reason about.
> 
> Changes of note:
> - We now use AbstractIdleService to manage the call order on the 
> SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
> - Many mesos-specific classes were moved to 
> `org.apache.aurora.scheduler.mesos`
> - Command line args used directly from `DriverFactoryImpl` are now properly 
> injected via `DriverSettings`
> - Untestable code around creating the driver is cordoned off to 
> `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
> constructor
> - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
> and `CommandLineDriverSettingsModule`
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
>   src/main/java/org/apache/aurora/scheduler/Driver.java 
> b07378f3772136177c81a65497152e9d6524b0a7 
>   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
> 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
>   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
> c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
>   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
> ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
>   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
> 453f22a8e97dbf535dd99598332e71e944ebf8d2 
>   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
> 8d306409bc382fbe5c53bb394d03c923b8f75889 
>   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
> 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
>   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
> 048740953629a950c136179c9b880b8dce8fa932 
>   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
> 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> feb70a9351fe804b6745304e4cbbf6d010c56579 
>   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
> 0b023a25292907b8211274f1533d89de7c4b7ce4 
>   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
> e656c7ec7349b54b07f8c734266445d1fc15fabb 
>   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
> 1a45d08c8854a335161476c8321c751f763dc12e 
>   src/main/java

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Kevin Sweeney 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60400
---

Ship it!



src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java


typo in blockUntilStopped



src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java


Technically it's dynamically-loaded.



src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java


Futures.getUnchecked



src/test/java/org/apache/aurora/scheduler/app/local/FakeMaster.java


Futures.getUnchecked


- Kevin Sweeney


On Nov. 7, 2014, 2:03 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27746/
> ---
> 
> (Updated Nov. 7, 2014, 2:03 p.m.)
> 
> 
> Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-920
> https://issues.apache.org/jira/browse/AURORA-920
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Apologies for the large diff - this is a thread that began with fixing a 
> SchedulerDriver-related race that would exist if we moved to an async event 
> bus.  Most of the green code is not new, but extracted from broken-apart 
> and/or deleted files.  However, i think this refactor makes the lifecycle of 
> the scheduler (and driver) easier to reason about.
> 
> Changes of note:
> - We now use AbstractIdleService to manage the call order on the 
> SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
> - Many mesos-specific classes were moved to 
> `org.apache.aurora.scheduler.mesos`
> - Command line args used directly from `DriverFactoryImpl` are now properly 
> injected via `DriverSettings`
> - Untestable code around creating the driver is cordoned off to 
> `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
> constructor
> - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
> and `CommandLineDriverSettingsModule`
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
>   src/main/java/org/apache/aurora/scheduler/Driver.java 
> b07378f3772136177c81a65497152e9d6524b0a7 
>   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
> 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
>   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
> c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
>   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
> ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
>   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
> 453f22a8e97dbf535dd99598332e71e944ebf8d2 
>   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
> 8d306409bc382fbe5c53bb394d03c923b8f75889 
>   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
> 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
>   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
> 048740953629a950c136179c9b880b8dce8fa932 
>   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
> 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> feb70a9351fe804b6745304e4cbbf6d010c56579 
>   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
> 0b023a25292907b8211274f1533d89de7c4b7ce4 
>   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
> e656c7ec7349b54b07f8c734266445d1fc15fabb 
>   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
> 1a45d08c8854a335161476c8321c751f763dc12e 
>   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
> 89140223f4f3acd02ade6fb95734744ef19d89bc 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
> 6d2ac494030018b100c1d01ce373c4896f2ed7d2 
>   src/main

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Maxim Khutornenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60398
---



src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java


LOG.info("Retrying for ... in ...")



src/main/java/org/apache/aurora/scheduler/mesos/Driver.java


is this really spelled as "blocK" with capital K?



src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java


Does it have to be public?



src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java


in(Singleton.class)?



src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java


Can you elaborate on the "rm -rf" effect? It's not clear how that happens.


- Maxim Khutornenko


On Nov. 7, 2014, 10:03 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27746/
> ---
> 
> (Updated Nov. 7, 2014, 10:03 p.m.)
> 
> 
> Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-920
> https://issues.apache.org/jira/browse/AURORA-920
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Apologies for the large diff - this is a thread that began with fixing a 
> SchedulerDriver-related race that would exist if we moved to an async event 
> bus.  Most of the green code is not new, but extracted from broken-apart 
> and/or deleted files.  However, i think this refactor makes the lifecycle of 
> the scheduler (and driver) easier to reason about.
> 
> Changes of note:
> - We now use AbstractIdleService to manage the call order on the 
> SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
> - Many mesos-specific classes were moved to 
> `org.apache.aurora.scheduler.mesos`
> - Command line args used directly from `DriverFactoryImpl` are now properly 
> injected via `DriverSettings`
> - Untestable code around creating the driver is cordoned off to 
> `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
> constructor
> - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
> and `CommandLineDriverSettingsModule`
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
>   src/main/java/org/apache/aurora/scheduler/Driver.java 
> b07378f3772136177c81a65497152e9d6524b0a7 
>   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
> 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
>   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
> c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
>   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
> ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
>   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
> 453f22a8e97dbf535dd99598332e71e944ebf8d2 
>   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
> 8d306409bc382fbe5c53bb394d03c923b8f75889 
>   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
> 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
>   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
> 048740953629a950c136179c9b880b8dce8fa932 
>   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
> 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> feb70a9351fe804b6745304e4cbbf6d010c56579 
>   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
> 0b023a25292907b8211274f1533d89de7c4b7ce4 
>   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
> e656c7ec7349b54b07f8c734266445d1fc15fabb 
>   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
> 1a45d08c8854a335161476c8321c751f763dc12e 
>   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
> 89140223f4f3acd02ade6fb95734744ef19d89bc 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> PRE-CREATION 
>   src/m

Re: Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/#review60402
---

Ship it!


Master (03cb0d1) is green with this patch.
  ./build-support/jenkins/build.sh

- Aurora ReviewBot


On Nov. 7, 2014, 10:03 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27746/
> ---
> 
> (Updated Nov. 7, 2014, 10:03 p.m.)
> 
> 
> Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-920
> https://issues.apache.org/jira/browse/AURORA-920
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Apologies for the large diff - this is a thread that began with fixing a 
> SchedulerDriver-related race that would exist if we moved to an async event 
> bus.  Most of the green code is not new, but extracted from broken-apart 
> and/or deleted files.  However, i think this refactor makes the lifecycle of 
> the scheduler (and driver) easier to reason about.
> 
> Changes of note:
> - We now use AbstractIdleService to manage the call order on the 
> SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
> - Many mesos-specific classes were moved to 
> `org.apache.aurora.scheduler.mesos`
> - Command line args used directly from `DriverFactoryImpl` are now properly 
> injected via `DriverSettings`
> - Untestable code around creating the driver is cordoned off to 
> `DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
> constructor
> - Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
> and `CommandLineDriverSettingsModule`
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
>   src/main/java/org/apache/aurora/scheduler/Driver.java 
> b07378f3772136177c81a65497152e9d6524b0a7 
>   src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
> 9cc04a84a37374ffca418e2ff767992ee23b9f3e 
>   src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
> c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
>   src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
> ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
>   src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
> 453f22a8e97dbf535dd99598332e71e944ebf8d2 
>   src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
> 8d306409bc382fbe5c53bb394d03c923b8f75889 
>   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
> 9ec3f412a9a9210279a2ce666bd99f61481ab92d 
>   src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
> 048740953629a950c136179c9b880b8dce8fa932 
>   src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
> 9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
>   src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
> feb70a9351fe804b6745304e4cbbf6d010c56579 
>   src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
> 0b023a25292907b8211274f1533d89de7c4b7ce4 
>   src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
> e656c7ec7349b54b07f8c734266445d1fc15fabb 
>   src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
> 1a45d08c8854a335161476c8321c751f763dc12e 
>   src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
> 89140223f4f3acd02ade6fb95734744ef19d89bc 
>   
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
> 6d2ac494030018b100c1d01ce373c4896f2ed7d2 
>   src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
> fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
>   src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
> 78a9670573261b2bdae0810c104ca7c1e4736ea8 
>   src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 
> cbb711fa555c94626963b4d9540a71e6a737abd8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java 
> 855573efea68ba4f29b3c7fbfb8da56bbdad352b 
>   src/test/java/org/apache/aurora/scheduler/DriverFactoryImplTest.java 
>

Review Request 27746: Simplify management of the driver lifecycle using AbstractidleService.

2014-11-07 Thread Bill Farner 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27746/
---

Review request for Aurora, Kevin Sweeney and Maxim Khutornenko.


Bugs: AURORA-920
https://issues.apache.org/jira/browse/AURORA-920


Repository: aurora


Description
---

Apologies for the large diff - this is a thread that began with fixing a 
SchedulerDriver-related race that would exist if we moved to an async event 
bus.  Most of the green code is not new, but extracted from broken-apart and/or 
deleted files.  However, i think this refactor makes the lifecycle of the 
scheduler (and driver) easier to reason about.

Changes of note:
- We now use AbstractIdleService to manage the call order on the 
SchedulerDriver (as well as HistoryPruner, but that is only loosely related).
- Many mesos-specific classes were moved to `org.apache.aurora.scheduler.mesos`
- Command line args used directly from `DriverFactoryImpl` are now properly 
injected via `DriverSettings`
- Untestable code around creating the driver is cordoned off to 
`DriverFactoryImpl`, which is a minimal shim over the `MesosSchedulerDriver` 
constructor
- Extracted modules that should not be used in tests: `LibMesosLoadingModule` 
and `CommandLineDriverSettingsModule`


Diffs
-

  config/legacy_untested_classes.txt f1d19210a722ee2c1838c4ffa0eec1f42d05c2d3 
  src/main/java/org/apache/aurora/scheduler/Driver.java 
b07378f3772136177c81a65497152e9d6524b0a7 
  src/main/java/org/apache/aurora/scheduler/DriverFactory.java 
9cc04a84a37374ffca418e2ff767992ee23b9f3e 
  src/main/java/org/apache/aurora/scheduler/MesosSchedulerImpl.java 
c424ecdaefb1b872adb4ef32a2aca088cee26f7f 
  src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 
ad07bcaa3a372efab1b6ba6165f5c44acf8de4cf 
  src/main/java/org/apache/aurora/scheduler/SchedulerLifecycle.java 
453f22a8e97dbf535dd99598332e71e944ebf8d2 
  src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 
8d306409bc382fbe5c53bb394d03c923b8f75889 
  src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
9ec3f412a9a9210279a2ce666bd99f61481ab92d 
  src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 
048740953629a950c136179c9b880b8dce8fa932 
  src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 
9bc42d3761d3fc72f009716cf087d4ac3c26dce4 
  src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 
feb70a9351fe804b6745304e4cbbf6d010c56579 
  src/main/java/org/apache/aurora/scheduler/async/JobUpdateHistoryPruner.java 
0b023a25292907b8211274f1533d89de7c4b7ce4 
  src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 
e656c7ec7349b54b07f8c734266445d1fc15fabb 
  src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 
1a45d08c8854a335161476c8321c751f763dc12e 
  src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 
89140223f4f3acd02ade6fb95734744ef19d89bc 
  
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
 PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/Driver.java PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverFactory.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/DriverSettings.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/LibMesosLoadingModule.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 
6d2ac494030018b100c1d01ce373c4896f2ed7d2 
  src/main/java/org/apache/aurora/scheduler/state/StateModule.java 
fe16d3a367b4b99b32a25d5ae55a3817a290e5e0 
  src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java 
78a9670573261b2bdae0810c104ca7c1e4736ea8 
  src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 
cbb711fa555c94626963b4d9540a71e6a737abd8 
  src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java 
855573efea68ba4f29b3c7fbfb8da56bbdad352b 
  src/test/java/org/apache/aurora/scheduler/DriverFactoryImplTest.java 
0f6192269cc122580869ae7717acad161fac6f1d 
  src/test/java/org/apache/aurora/scheduler/DriverTest.java 
a96dd87421c32e4da4da4c953dda4c6be4e8e592 
  src/test/java/org/apache/aurora/scheduler/MesosSchedulerImplTest.java 
8dd908e69c71e96d03806e0c4ceb00dab81d756f 
  src/test/java/org/apache/aurora/scheduler/MesosTaskFactoryImplTest.java 
aa026cf3b6ace5e0aaf9b239aff0c2f3210a0883 
  src/test/java/org/apache/aurora/scheduler/SchedulerLifecycleTest.java 
986160133a3471fc36cad2615c2aa1a7d8983b5a 
  src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 
713797182e22956640eeffb796e0a0ff7ec3ddc9 
  src/test/java/org/apache/aurora/