from:"Mehrdad Nurolahzade"

Review Request 48082: AURORA-1624 Make 'tier' required and remove support for 'production' flag in Job configuration - New thrift API for retrieving tier configuration

2016-05-31 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48082/
---

Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Repository: aurora


Description
---

AURORA-1624 Make 'tier' required and remove support for 'production' flag in 
Job configuration - New thrift API for retrieving tier configuration


Diffs
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
ed94f249d85ac0e438213924c777cf7c029a119a 
  src/main/java/org/apache/aurora/scheduler/TierManager.java 
b96189be0ada1623665c2bff2550c1d72d7bc3dd 
  src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
e431b58c933886f46c095240704d3eb0ceea2d80 
  src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
0d4f04403ec20c21b7cfacd706557cd191579f0a 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
4d032b96d55dd0b92fab338220e66628e38cbb11 
  src/main/resources/org/apache/aurora/scheduler/tiers.json 
f724c5ad03a1315bc55dac35d98fdef45625e017 
  src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
95174bb7454a9e7c075ebaa6a4f84bf55fbc2652 
  src/test/java/org/apache/aurora/scheduler/TierModuleTest.java 
6b4e7a0d5e64d0632c66273ad516f737b2ef4a92 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
2122f744089adaee24328a634a2f786d1ef9720f 

Diff: https://reviews.apache.org/r/48082/diff/


Testing
---

Manual/Explorative: Invoked from CLI-side to see if the new API call works as 
intended
Integration: ./build-support/jenkins/build.sh
E2E: ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh


Thanks,

Mehrdad Nurolahzade

Re: Review Request 48082: AURORA-1624 Make 'tier' required and remove support for 'production' flag in Job configuration - New thrift API for retrieving tier configuration

2016-06-06 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48082/
---

(Updated June 6, 2016, 7:13 p.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Changes
---

(1) Applied suggested refactorings
(2) Raised AURORA-1708 for deprecation of 'production' flag


Repository: aurora


Description
---

AURORA-1624 Make 'tier' required and remove support for 'production' flag in 
Job configuration - New thrift API for retrieving tier configuration


Diffs (updated)
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
ed94f249d85ac0e438213924c777cf7c029a119a 
  src/main/java/org/apache/aurora/scheduler/TierManager.java 
b96189be0ada1623665c2bff2550c1d72d7bc3dd 
  src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
e431b58c933886f46c095240704d3eb0ceea2d80 
  src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
0d4f04403ec20c21b7cfacd706557cd191579f0a 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
4d032b96d55dd0b92fab338220e66628e38cbb11 
  src/main/resources/org/apache/aurora/scheduler/tiers.json 
f724c5ad03a1315bc55dac35d98fdef45625e017 
  src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
95174bb7454a9e7c075ebaa6a4f84bf55fbc2652 
  src/test/java/org/apache/aurora/scheduler/TierModuleTest.java 
6b4e7a0d5e64d0632c66273ad516f737b2ef4a92 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
2122f744089adaee24328a634a2f786d1ef9720f 

Diff: https://reviews.apache.org/r/48082/diff/


Testing
---

Manual/Explorative: Invoked from CLI-side to see if the new API call works as 
intended
Integration: ./build-support/jenkins/build.sh
E2E: ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh


Thanks,

Mehrdad Nurolahzade

Re: Review Request 48082: AURORA-1624 Make 'tier' required and remove support for 'production' flag in Job configuration - New thrift API for retrieving tier configuration

2016-06-06 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48082/
---

(Updated June 7, 2016, 4:11 a.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Changes
---

(1) Applied suggested refactorings
(2) Updated release notes


Repository: aurora


Description
---

AURORA-1624 Make 'tier' required and remove support for 'production' flag in 
Job configuration - New thrift API for retrieving tier configuration


Diffs (updated)
-

  RELEASE-NOTES.md 7d47cf63a5529b43ca06cfb0b9e171a90a56f7f8 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
ed94f249d85ac0e438213924c777cf7c029a119a 
  src/main/java/org/apache/aurora/scheduler/TierInfo.java 
ac8901fe90d57c541829247fa9fa0895eb019e87 
  src/main/java/org/apache/aurora/scheduler/TierManager.java 
b96189be0ada1623665c2bff2550c1d72d7bc3dd 
  src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
e431b58c933886f46c095240704d3eb0ceea2d80 
  src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
0d4f04403ec20c21b7cfacd706557cd191579f0a 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
4d032b96d55dd0b92fab338220e66628e38cbb11 
  src/main/resources/org/apache/aurora/scheduler/tiers.json 
f724c5ad03a1315bc55dac35d98fdef45625e017 
  src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
95174bb7454a9e7c075ebaa6a4f84bf55fbc2652 
  src/test/java/org/apache/aurora/scheduler/TierModuleTest.java 
6b4e7a0d5e64d0632c66273ad516f737b2ef4a92 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
2122f744089adaee24328a634a2f786d1ef9720f 

Diff: https://reviews.apache.org/r/48082/diff/


Testing
---

Manual/Explorative: Invoked from CLI-side to see if the new API call works as 
intended
Integration: ./build-support/jenkins/build.sh
E2E: ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh


Thanks,

Mehrdad Nurolahzade

Re: Review Request 48082: AURORA-1624 New thrift API for retrieving tier configuration

2016-06-07 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48082/
---

(Updated June 7, 2016, 8:17 p.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Changes
---

(1) Applied suggested refactoring
(2) Updated release notes


Summary (updated)
-

AURORA-1624 New thrift API for retrieving tier configuration


Repository: aurora


Description (updated)
---

AURORA-1624 Make 'tier' required and remove support for 'production' flag in 
Job configuration
(1) New thrift API for retrieving tier configuration
(2) `production` field in `TaskConfig` thrift struct is deprecated (will be 
removed in next release)
(3) `default` property in `tiers.json` to specify the name of the default tier


Diffs (updated)
-

  RELEASE-NOTES.md 96bc3ca4bfd85c3e307f186ab24252c297ba336c 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
ed94f249d85ac0e438213924c777cf7c029a119a 
  src/main/java/org/apache/aurora/scheduler/TierInfo.java 
ac8901fe90d57c541829247fa9fa0895eb019e87 
  src/main/java/org/apache/aurora/scheduler/TierManager.java 
b96189be0ada1623665c2bff2550c1d72d7bc3dd 
  src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
e431b58c933886f46c095240704d3eb0ceea2d80 
  src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
0d4f04403ec20c21b7cfacd706557cd191579f0a 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
4d032b96d55dd0b92fab338220e66628e38cbb11 
  src/main/resources/org/apache/aurora/scheduler/tiers.json 
f724c5ad03a1315bc55dac35d98fdef45625e017 
  src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
95174bb7454a9e7c075ebaa6a4f84bf55fbc2652 
  src/test/java/org/apache/aurora/scheduler/TierModuleTest.java 
6b4e7a0d5e64d0632c66273ad516f737b2ef4a92 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
2122f744089adaee24328a634a2f786d1ef9720f 

Diff: https://reviews.apache.org/r/48082/diff/


Testing
---

Manual/Explorative: Invoked from CLI-side to see if the new API call works as 
intended
Integration: ./build-support/jenkins/build.sh
E2E: ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh


Thanks,

Mehrdad Nurolahzade

Re: Review Request 48082: AURORA-1624 New thrift API for retrieving tier configuration

2016-06-07 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48082/
---

(Updated June 7, 2016, 8:40 p.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Changes
---

+ticket


Bugs: AURORA-1624
https://issues.apache.org/jira/browse/AURORA-1624


Repository: aurora


Description
---

AURORA-1624 Make 'tier' required and remove support for 'production' flag in 
Job configuration
(1) New thrift API for retrieving tier configuration
(2) `production` field in `TaskConfig` thrift struct is deprecated (will be 
removed in next release)
(3) `default` property in `tiers.json` to specify the name of the default tier


Diffs
-

  RELEASE-NOTES.md 96bc3ca4bfd85c3e307f186ab24252c297ba336c 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
ed94f249d85ac0e438213924c777cf7c029a119a 
  src/main/java/org/apache/aurora/scheduler/TierInfo.java 
ac8901fe90d57c541829247fa9fa0895eb019e87 
  src/main/java/org/apache/aurora/scheduler/TierManager.java 
b96189be0ada1623665c2bff2550c1d72d7bc3dd 
  src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
e431b58c933886f46c095240704d3eb0ceea2d80 
  src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
0d4f04403ec20c21b7cfacd706557cd191579f0a 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
4d032b96d55dd0b92fab338220e66628e38cbb11 
  src/main/resources/org/apache/aurora/scheduler/tiers.json 
f724c5ad03a1315bc55dac35d98fdef45625e017 
  src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
95174bb7454a9e7c075ebaa6a4f84bf55fbc2652 
  src/test/java/org/apache/aurora/scheduler/TierModuleTest.java 
6b4e7a0d5e64d0632c66273ad516f737b2ef4a92 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
2122f744089adaee24328a634a2f786d1ef9720f 

Diff: https://reviews.apache.org/r/48082/diff/


Testing
---

Manual/Explorative: Invoked from CLI-side to see if the new API call works as 
intended
Integration: ./build-support/jenkins/build.sh
E2E: ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh


Thanks,

Mehrdad Nurolahzade

Review Request 48559: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - Backfill portion

2016-06-10 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48559/
---

Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Repository: aurora


Description
---

AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
Job configuration - Backfill portion


Diffs
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
3e6daf53dd563dd7a2d494cc95e9a0aba0b6 
  src/main/java/org/apache/aurora/scheduler/TierManager.java 
af54cab73a80a5120b1a77fd985dfbaf568d786c 
  src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
3ea0992eb0a9930a4db9eb4b7fcab82689495c1f 
  
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
 0e9562020c298e685e6c2efd18933818b03a5000 
  
src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java 
d08873c88f159eb65b582840b48b7ff604862c31 
  
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V006_PopulateTierField.java
 PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java 
c981a05e61cb053a05144c702c9ffafeb0af8260 
  src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java 
8eed1fc680b0c4fb27d8a353b7f804ae09058156 
  src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java 
0a307fe8d8238c23a526d5c3ee500e1de0761703 
  src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
d4b71f8dbb674384ccbbd9e76f510d127e480e32 
  
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
 2e322d217fc9dc75c51b57607a5547745206fb9f 
  src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java 
e870087e3d47906559410ff76515457f4ff99ff5 
  src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java 
be1132b439948104458efdc82a6bbee43c20c4fd 
  
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java 
e0cf602ead1530301b09eff60287b8fa48be63e8 
  src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java 
0a2cd3d5b01c389f99fca169227aac35436d474b 
  src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java 
4f8158546f3eba8f79d653ad7a30f83d66cbce83 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 ecdc62ae3b21b73b6a6af80bb9855867a7e965e0 

Diff: https://reviews.apache.org/r/48559/diff/


Testing
---

Manual under Vagrant:
- Deployed old scheduler (with tier backfill support), created a job without 
tier, upgraded scheduler, noticed that tier has been backfilled
- Tried the above scenario with both -use_beta_db_task_store=true and 
-use_beta_db_task_store=false configuration flags
- Verified that if tier is already set it would not be altered
- Verified that it works both when production = 'true' and production = 'false'

End to End:
```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

...

*** OK (All tests passed) ***

mesos-master start/running, process 26886
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real17m53.514s
user0m1.443s
sys 0m0.624s
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 48559: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - Backfill portion

2016-06-14 Thread Mehrdad Nurolahzade

/scheduler/resources/ResourceManagerTest.java 
14ac54713acf69fc40807b8cf7345b7a043e1ad9 
  
src/test/java/org/apache/aurora/scheduler/scheduling/RescheduleCalculatorImplTest.java
 2d34729e8f371703ba250b8e82d82c8a2504fcac 
  
src/test/java/org/apache/aurora/scheduler/scheduling/TaskSchedulerImplTest.java 
fba427bd327e7f63b640c8b8753bfdeec3ee31e7 
  src/test/java/org/apache/aurora/scheduler/sla/SlaTestUtil.java 
78f440f7546de9ed6842cb51db02b3bddc9a74ff 
  
src/test/java/org/apache/aurora/scheduler/state/MaintenanceControllerImplTest.java
 94f5ca565476f62d72879837a0e7dafabcf30432 
  src/test/java/org/apache/aurora/scheduler/state/StateManagerImplTest.java 
fbcc4003f7647580b859f4dc08a16c3e471a8f18 
  src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
b4d27f69ad5d4cce03da9f04424dc35d30e8af29 
  src/test/java/org/apache/aurora/scheduler/state/TaskStateMachineTest.java 
c5193238db5cb65373f7cb69e6b38b2e3b051dae 
  src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java 
19c1f8e949e31a2338a5cbbbf15228230377a9f5 
  
src/test/java/org/apache/aurora/scheduler/storage/AbstractCronJobStoreTest.java 
c316e497a34a45c7ada2ca83a1115e826c0f572f 
  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
b1593f682f48ea66339bc2372de3e4f14e40be32 
  src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java 
e870087e3d47906559410ff76515457f4ff99ff5 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbAttributeStoreTest.java 
f47f4a8a492fb43bacd909dc520256ed028531dd 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 
08530397ff75081bde6f07f9d53317b5486e0da4 
  
src/test/java/org/apache/aurora/scheduler/storage/db/RowGarbageCollectorTest.java
 3e5296e40ba63dc06a4720f1ff2c1ff046613ea2 
  src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java 
73440517cddda643c0b84cc04cb8463cdea2da28 
  src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java 
be1132b439948104458efdc82a6bbee43c20c4fd 
  
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java 
e0cf602ead1530301b09eff60287b8fa48be63e8 
  src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java 
0a2cd3d5b01c389f99fca169227aac35436d474b 
  
src/test/java/org/apache/aurora/scheduler/storage/log/WriteAheadStorageTest.java
 13726cc11ab09cd4995233d9d31811b97b065275 
  
src/test/java/org/apache/aurora/scheduler/storage/mem/StorageTransactionTest.java
 25f34e2bc26c6d4754c1591fad7f2165dd465d32 
  src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java 
4f8158546f3eba8f79d653ad7a30f83d66cbce83 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
6f355d6e66c05651fa9b13356dd81b45bee52adc 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 ecdc62ae3b21b73b6a6af80bb9855867a7e965e0 
  src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 
a54d169caebfc211035386f64169ecd983e378d9 
  src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
56c94b5caf414861212f673a27b84d46c07332e6 
  
src/test/java/org/apache/aurora/scheduler/updater/JobUpdateEventSubscriberTest.java
 36f2c657c05a87e78a11a1b0be5779dfd6511ee5 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
e157c0dfde5efc418448e138aa008ade742fe816 
  src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
e5935f65924e7d9a2491cac8f4c1f575ec657776 

Diff: https://reviews.apache.org/r/48559/diff/


Testing
---

Manual under Vagrant:
- Deployed old scheduler (with tier backfill support), created a job without 
tier, upgraded scheduler, noticed that tier has been backfilled
- Tried the above scenario with both -use_beta_db_task_store=true and 
-use_beta_db_task_store=false configuration flags
- Verified that if tier is already set it would not be altered
- Verified that it works both when production = 'true' and production = 'false'

End to End:
```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

...

*** OK (All tests passed) ***

mesos-master start/running, process 26886
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real17m53.514s
user0m1.443s
sys 0m0.624s
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 48559: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - Backfill portion

2016-06-14 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48559/#review137538
---




src/main/java/org/apache/aurora/scheduler/storage/db/migration/V006_PopulateTierField.java
 (lines 22 - 23)
<https://reviews.apache.org/r/48559/#comment202685>

Hardcoding tier names here can be problematic if someone decides to 
customize their tier configuration file.

I tried looking up tier names dynamically through ```TierManager``` but 
then realized that migration scripts are instantiated through reflection by 
ibatis and Guice injection did not work on them (or I don't know enough about 
Guice to make it work).

Any suggestions?


- Mehrdad Nurolahzade


On June 14, 2016, 9:46 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48559/
> ---
> 
> (Updated June 14, 2016, 9:46 a.m.)
> 
> 
> Review request for Aurora and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - Backfill portion
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 
> f4f8d0037751c9c2096747264c19f6292461b308 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
> 65f5edce74077f52e98f110fcd17b2f12d673f81 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 6fd9ee21cf8e0e42e73a68cdf2d231d581278aae 
>   src/jmh/java/org/apache/aurora/benchmark/Tasks.java 
> e548a09d94b1c6d550f9beec2b7120b64d576f20 
>   src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
> 7497fb96a578cff64b6484c587e29ec464969e84 
>   src/main/java/org/apache/aurora/scheduler/TierManager.java 
> af54cab73a80a5120b1a77fd985dfbaf568d786c 
>   src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> 3ea0992eb0a9930a4db9eb4b7fcab82689495c1f 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
>  0e9562020c298e685e6c2efd18933818b03a5000 
>   
> src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java
>  d08873c88f159eb65b582840b48b7ff604862c31 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V006_PopulateTierField.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java 
> c981a05e61cb053a05144c702c9ffafeb0af8260 
>   
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java 
> 8eed1fc680b0c4fb27d8a353b7f804ae09058156 
>   src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java 
> 0a307fe8d8238c23a526d5c3ee500e1de0761703 
>   src/test/java/org/apache/aurora/codec/ThriftBinaryCodecTest.java 
> 999ef064737c3d8a3d7610b40c13736f51742edd 
>   src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 
> 05cd78f4c7c7d8dd6eeb6f2f9a3e8f7a167f274d 
>   src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
> d4b71f8dbb674384ccbbd9e76f510d127e480e32 
>   src/test/java/org/apache/aurora/scheduler/TierModuleTest.java 
> 58d95dcdf31bc920ca1f8822baccc6c37b66e739 
>   src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 
> 8c8c793813e84bf7ef741f9b6b4ae1e759be1b82 
>   src/test/java/org/apache/aurora/scheduler/base/JobsTest.java 
> 13f656f241a8a9a3d339f4053f165070c2669ef3 
>   src/test/java/org/apache/aurora/scheduler/base/TasksTest.java 
> 935622bef38cc3d399e8b5b1db84fa21c79c78e1 
>   
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
>  2e322d217fc9dc75c51b57607a5547745206fb9f 
>   
> src/test/java/org/apache/aurora/scheduler/cron/quartz/AuroraCronJobTest.java 
> 5c64ff2994e200b3453603ac5470e8e152cebc55 
>   src/test/java/org/apache/aurora/scheduler/cron/quartz/CronIT.java 
> 1c0a3fa84874d7bc185b78f13d2664cb4d8dd72f 
>   
> src/test/java/org/apache/aurora/scheduler/cron/quartz/CronJobManagerImplTest.java
>  81440f5689f9538a4c7a9e6700bf03ca89c4ba85 
>   src/test/java/org/apache/aurora/scheduler/cron/quartz/QuartzTestUtil.java 
> 3c5ecd698557cafdf8eeacdc472589a379018896 
>   src/test/java/org/apache/aurora/scheduler/events/WebhookTest.java 
> 488eefd14c3e67a41a75c809397c8d19f83cc08a 
>   src/test/java/org/apache/aurora/scheduler/http/MaintenanceTest.java 
> f94b58b77b7c6ce824914af7e1147e73ad5a7eed 
>   src/test/java/org/apache/aurora/scheduler/http/api/ApiBetaTest.java 
> ccef4ab930a7b7e1c10d611f0852aa65de82e

Re: Review Request 48559: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - Backfill portion

2016-06-14 Thread Mehrdad Nurolahzade

test/java/org/apache/aurora/scheduler/sla/SlaTestUtil.java 
78f440f7546de9ed6842cb51db02b3bddc9a74ff 
  
src/test/java/org/apache/aurora/scheduler/state/MaintenanceControllerImplTest.java
 94f5ca565476f62d72879837a0e7dafabcf30432 
  src/test/java/org/apache/aurora/scheduler/state/StateManagerImplTest.java 
fbcc4003f7647580b859f4dc08a16c3e471a8f18 
  src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java 
b4d27f69ad5d4cce03da9f04424dc35d30e8af29 
  src/test/java/org/apache/aurora/scheduler/state/TaskStateMachineTest.java 
c5193238db5cb65373f7cb69e6b38b2e3b051dae 
  src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java 
19c1f8e949e31a2338a5cbbbf15228230377a9f5 
  
src/test/java/org/apache/aurora/scheduler/storage/AbstractCronJobStoreTest.java 
c316e497a34a45c7ada2ca83a1115e826c0f572f 
  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
b1593f682f48ea66339bc2372de3e4f14e40be32 
  src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java 
e870087e3d47906559410ff76515457f4ff99ff5 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbAttributeStoreTest.java 
f47f4a8a492fb43bacd909dc520256ed028531dd 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 
08530397ff75081bde6f07f9d53317b5486e0da4 
  
src/test/java/org/apache/aurora/scheduler/storage/db/RowGarbageCollectorTest.java
 3e5296e40ba63dc06a4720f1ff2c1ff046613ea2 
  src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java 
73440517cddda643c0b84cc04cb8463cdea2da28 
  src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java 
be1132b439948104458efdc82a6bbee43c20c4fd 
  
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java 
e0cf602ead1530301b09eff60287b8fa48be63e8 
  src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java 
0a2cd3d5b01c389f99fca169227aac35436d474b 
  
src/test/java/org/apache/aurora/scheduler/storage/log/WriteAheadStorageTest.java
 13726cc11ab09cd4995233d9d31811b97b065275 
  
src/test/java/org/apache/aurora/scheduler/storage/mem/StorageTransactionTest.java
 25f34e2bc26c6d4754c1591fad7f2165dd465d32 
  src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java 
4f8158546f3eba8f79d653ad7a30f83d66cbce83 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
6f355d6e66c05651fa9b13356dd81b45bee52adc 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 ecdc62ae3b21b73b6a6af80bb9855867a7e965e0 
  src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 
a54d169caebfc211035386f64169ecd983e378d9 
  src/test/java/org/apache/aurora/scheduler/updater/AddTaskTest.java 
56c94b5caf414861212f673a27b84d46c07332e6 
  
src/test/java/org/apache/aurora/scheduler/updater/JobUpdateEventSubscriberTest.java
 36f2c657c05a87e78a11a1b0be5779dfd6511ee5 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
e157c0dfde5efc418448e138aa008ade742fe816 
  src/test/java/org/apache/aurora/scheduler/updater/KillTaskTest.java 
e5935f65924e7d9a2491cac8f4c1f575ec657776 

Diff: https://reviews.apache.org/r/48559/diff/


Testing
---

Manual under Vagrant:
- Deployed old scheduler (with tier backfill support), created a job without 
tier, upgraded scheduler, noticed that tier has been backfilled
- Tried the above scenario with both -use_beta_db_task_store=true and 
-use_beta_db_task_store=false configuration flags
- Verified that if tier is already set it would not be altered
- Verified that it works both when production = 'true' and production = 'false'

End to End:
```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

...

*** OK (All tests passed) ***

mesos-master start/running, process 26886
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real17m53.514s
user0m1.443s
sys 0m0.624s
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 48559: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - Backfill portion

2016-06-14 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48559/#review137553
---



@ReviewBot retry

- Mehrdad Nurolahzade


On June 14, 2016, 10:12 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48559/
> ---
> 
> (Updated June 14, 2016, 10:12 a.m.)
> 
> 
> Review request for Aurora and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - Backfill portion
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 
> f4f8d0037751c9c2096747264c19f6292461b308 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
> 65f5edce74077f52e98f110fcd17b2f12d673f81 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 6fd9ee21cf8e0e42e73a68cdf2d231d581278aae 
>   src/jmh/java/org/apache/aurora/benchmark/Tasks.java 
> e548a09d94b1c6d550f9beec2b7120b64d576f20 
>   src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
> 7497fb96a578cff64b6484c587e29ec464969e84 
>   src/main/java/org/apache/aurora/scheduler/TierManager.java 
> af54cab73a80a5120b1a77fd985dfbaf568d786c 
>   src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> 3ea0992eb0a9930a4db9eb4b7fcab82689495c1f 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
>  0e9562020c298e685e6c2efd18933818b03a5000 
>   
> src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java
>  d08873c88f159eb65b582840b48b7ff604862c31 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V006_PopulateTierField.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java 
> c981a05e61cb053a05144c702c9ffafeb0af8260 
>   
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java 
> 8eed1fc680b0c4fb27d8a353b7f804ae09058156 
>   src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java 
> 0a307fe8d8238c23a526d5c3ee500e1de0761703 
>   src/test/java/org/apache/aurora/codec/ThriftBinaryCodecTest.java 
> 999ef064737c3d8a3d7610b40c13736f51742edd 
>   src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 
> 05cd78f4c7c7d8dd6eeb6f2f9a3e8f7a167f274d 
>   src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
> d4b71f8dbb674384ccbbd9e76f510d127e480e32 
>   src/test/java/org/apache/aurora/scheduler/TierModuleTest.java 
> 58d95dcdf31bc920ca1f8822baccc6c37b66e739 
>   src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 
> 8c8c793813e84bf7ef741f9b6b4ae1e759be1b82 
>   src/test/java/org/apache/aurora/scheduler/base/JobsTest.java 
> 13f656f241a8a9a3d339f4053f165070c2669ef3 
>   src/test/java/org/apache/aurora/scheduler/base/TasksTest.java 
> 935622bef38cc3d399e8b5b1db84fa21c79c78e1 
>   
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
>  2e322d217fc9dc75c51b57607a5547745206fb9f 
>   
> src/test/java/org/apache/aurora/scheduler/cron/quartz/AuroraCronJobTest.java 
> 5c64ff2994e200b3453603ac5470e8e152cebc55 
>   src/test/java/org/apache/aurora/scheduler/cron/quartz/CronIT.java 
> 1c0a3fa84874d7bc185b78f13d2664cb4d8dd72f 
>   
> src/test/java/org/apache/aurora/scheduler/cron/quartz/CronJobManagerImplTest.java
>  81440f5689f9538a4c7a9e6700bf03ca89c4ba85 
>   src/test/java/org/apache/aurora/scheduler/cron/quartz/QuartzTestUtil.java 
> 3c5ecd698557cafdf8eeacdc472589a379018896 
>   src/test/java/org/apache/aurora/scheduler/events/WebhookTest.java 
> 488eefd14c3e67a41a75c809397c8d19f83cc08a 
>   src/test/java/org/apache/aurora/scheduler/http/MaintenanceTest.java 
> f94b58b77b7c6ce824914af7e1147e73ad5a7eed 
>   src/test/java/org/apache/aurora/scheduler/http/api/ApiBetaTest.java 
> ccef4ab930a7b7e1c10d611f0852aa65de82e726 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
> 58785bfa37ff214f26e9f94d836e6df40e411c3b 
>   
> src/test/java/org/apache/aurora/scheduler/preemptor/PreemptionVictimFilterTest.java
>  7eb1714d14581a6ab25e85d36a1f3e973380c536 
>   
> src/test/java/org/apache/aurora/scheduler/pruning/TaskHistoryPrunerTest.java 
> 99c27e8012f10a67ce5f1b84d258e7a5608995c7 
>   src/test/java/org/apache/aurora/scheduler/quota/QuotaManagerImplTest.java 
> 2e97a3361feaed71e4f39cbd27cf5afb7d919

Re: Review Request 48559: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - Backfill portion

2016-06-15 Thread Mehrdad Nurolahzade



> On June 15, 2016, 10:41 a.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java, line 14
> > <https://reviews.apache.org/r/48559/diff/3/?file=1418765#file1418765line14>
> >
> > Not sure why this class has been moved but if you want to keep it 
> > please carve it out into a separate patch. Combining functional changes 
> > with moving files around inflates the diff size and makes it harder to 
> > understand the commit in retrospect.

I can revert this package move refactoring. This was required by the 
refactorings to ```TaskTestUtil```. 
I can alternatively make the constructor of ```TierManagerImpl``` public 
(instead of moving them both to the same package).


> On June 15, 2016, 10:41 a.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/db/migration/V006_PopulateTierField.java,
> >  lines 50-52
> > <https://reviews.apache.org/r/48559/diff/3/?file=1418768#file1418768line50>
> >
> > This should be unnecessary as zero is the default value for 
> > 'production'.

Isn't this required to resolve the inconsistencies between the choice of 
```production``` and ```tier``` in already scheduled jobs?
For example, if tier has been set to ```revocable``` and production to 
```true``` in a job submitted to the old scheduler, shouldn't we resolve this 
by resetting ```production``` to ```false```?


> On June 15, 2016, 10:41 a.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/TierManager.java, lines 125-127
> > <https://reviews.apache.org/r/48559/diff/3/?file=1418764#file1418764line125>
> >
> > Reiterating my previous comment: can this (and other places checking 
> > for tier presence) be simplified now that the tier is a required field?

This logic is used by ```ThriftBackfill``` to set ```tier``` when it is not 
already set.
Shouldn't this be removed in the next release when we drop the back fill logic?


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48559/#review137763
---


On June 14, 2016, 10:12 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48559/
> ---
> 
> (Updated June 14, 2016, 10:12 a.m.)
> 
> 
> Review request for Aurora and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - Backfill portion
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 
> f4f8d0037751c9c2096747264c19f6292461b308 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
> 65f5edce74077f52e98f110fcd17b2f12d673f81 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 6fd9ee21cf8e0e42e73a68cdf2d231d581278aae 
>   src/jmh/java/org/apache/aurora/benchmark/Tasks.java 
> e548a09d94b1c6d550f9beec2b7120b64d576f20 
>   src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
> 7497fb96a578cff64b6484c587e29ec464969e84 
>   src/main/java/org/apache/aurora/scheduler/TierManager.java 
> af54cab73a80a5120b1a77fd985dfbaf568d786c 
>   src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> 3ea0992eb0a9930a4db9eb4b7fcab82689495c1f 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
>  0e9562020c298e685e6c2efd18933818b03a5000 
>   
> src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java
>  d08873c88f159eb65b582840b48b7ff604862c31 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V006_PopulateTierField.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java 
> c981a05e61cb053a05144c702c9ffafeb0af8260 
>   
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java 
> 8eed1fc680b0c4fb27d8a353b7f804ae09058156 
>   src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java 
> 0a307fe8d8238c23a526d5c3ee500e1de0761703 
>   src/test/java/org/apache/aurora/codec/ThriftBinaryCodecTest.java 
> 999ef064737c3d8a3d7610b40c13736f51742edd 
>   src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 
> 05cd78f4c7c7d8dd6eeb6f2f9a3e8f7a167f274d 
>   src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
> d4b71f8dbb674384ccbbd9e76f510d127

Re: Review Request 48559: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - Backfill portion

2016-06-16 Thread Mehrdad Nurolahzade



> On June 15, 2016, 10:41 a.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/db/migration/V006_PopulateTierField.java,
> >  lines 50-52
> > <https://reviews.apache.org/r/48559/diff/3/?file=1418768#file1418768line50>
> >
> > This should be unnecessary as zero is the default value for 
> > 'production'.
> 
> Mehrdad Nurolahzade wrote:
> Isn't this required to resolve the inconsistencies between the choice of 
> ```production``` and ```tier``` in already scheduled jobs?
> For example, if tier has been set to ```revocable``` and production to 
> ```true``` in a job submitted to the old scheduler, shouldn't we resolve this 
> by resetting ```production``` to ```false```?

Actually, thinking twice about this, do we even need a downgrade script?
I mean, we are already modifying ```production``` based on choice of ```tier``` 
for scheduled jobs, so it should not be a concern when downgrading scheduler, 
right?


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48559/#review137763
-----------


On June 14, 2016, 10:12 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48559/
> ---
> 
> (Updated June 14, 2016, 10:12 a.m.)
> 
> 
> Review request for Aurora and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - Backfill portion
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 
> f4f8d0037751c9c2096747264c19f6292461b308 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
> 65f5edce74077f52e98f110fcd17b2f12d673f81 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 6fd9ee21cf8e0e42e73a68cdf2d231d581278aae 
>   src/jmh/java/org/apache/aurora/benchmark/Tasks.java 
> e548a09d94b1c6d550f9beec2b7120b64d576f20 
>   src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
> 7497fb96a578cff64b6484c587e29ec464969e84 
>   src/main/java/org/apache/aurora/scheduler/TierManager.java 
> af54cab73a80a5120b1a77fd985dfbaf568d786c 
>   src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> 3ea0992eb0a9930a4db9eb4b7fcab82689495c1f 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
>  0e9562020c298e685e6c2efd18933818b03a5000 
>   
> src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java
>  d08873c88f159eb65b582840b48b7ff604862c31 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V006_PopulateTierField.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java 
> c981a05e61cb053a05144c702c9ffafeb0af8260 
>   
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java 
> 8eed1fc680b0c4fb27d8a353b7f804ae09058156 
>   src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java 
> 0a307fe8d8238c23a526d5c3ee500e1de0761703 
>   src/test/java/org/apache/aurora/codec/ThriftBinaryCodecTest.java 
> 999ef064737c3d8a3d7610b40c13736f51742edd 
>   src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 
> 05cd78f4c7c7d8dd6eeb6f2f9a3e8f7a167f274d 
>   src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
> d4b71f8dbb674384ccbbd9e76f510d127e480e32 
>   src/test/java/org/apache/aurora/scheduler/TierModuleTest.java 
> 58d95dcdf31bc920ca1f8822baccc6c37b66e739 
>   src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 
> 8c8c793813e84bf7ef741f9b6b4ae1e759be1b82 
>   src/test/java/org/apache/aurora/scheduler/base/JobsTest.java 
> 13f656f241a8a9a3d339f4053f165070c2669ef3 
>   src/test/java/org/apache/aurora/scheduler/base/TasksTest.java 
> 935622bef38cc3d399e8b5b1db84fa21c79c78e1 
>   
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
>  2e322d217fc9dc75c51b57607a5547745206fb9f 
>   
> src/test/java/org/apache/aurora/scheduler/cron/quartz/AuroraCronJobTest.java 
> 5c64ff2994e200b3453603ac5470e8e152cebc55 
>   src/test/java/org/apache/aurora/scheduler/cron/quartz/CronIT.java 
> 1c0a3fa84874d7bc185b78f13d2664cb4d8dd72f 
>   
> src/test/java/org/apache/aurora/scheduler/cron/quartz/CronJobManagerImplTest.java
>  81440f5689f9538a4c7a9e6700bf03ca89c4ba85 
>   src/test/java/org/apach

Review Request 48796: AURORA-1458 Add tier into the UI "show config" summary

2016-06-16 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48796/
---

Review request for Aurora and Maxim Khutornenko.


Repository: aurora


Description
---

AURORA-1458 Add tier into the UI "show config" summary


Diffs
-

  src/main/resources/scheduler/assets/configSummary.html 
86a87ab710312969c31802492d856b04f07c276d 
  src/main/resources/scheduler/assets/js/services.js 
b3a0a994707972a9a19a7f4079dad67fe3136c1c 

Diff: https://reviews.apache.org/r/48796/diff/


Testing
---

```
./build-support/jenkins/build.sh

+ date
Thu Jun 16 08:57:08 PDT 2016
+ ./gradlew -Pq clean build

...

09:03:14 00:03   [complete]
   SUCCESS
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 48559: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - Backfill portion

2016-06-16 Thread Mehrdad Nurolahzade



> On June 15, 2016, 10:41 a.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/db/migration/V006_PopulateTierField.java,
> >  lines 50-52
> > <https://reviews.apache.org/r/48559/diff/3/?file=1418768#file1418768line50>
> >
> > This should be unnecessary as zero is the default value for 
> > 'production'.
> 
> Mehrdad Nurolahzade wrote:
> Isn't this required to resolve the inconsistencies between the choice of 
> ```production``` and ```tier``` in already scheduled jobs?
> For example, if tier has been set to ```revocable``` and production to 
> ```true``` in a job submitted to the old scheduler, shouldn't we resolve this 
> by resetting ```production``` to ```false```?
> 
> Mehrdad Nurolahzade wrote:
> Actually, thinking twice about this, do we even need a downgrade script?
> I mean, we are already modifying ```production``` based on choice of 
> ```tier``` for scheduled jobs, so it should not be a concern when downgrading 
> scheduler, right?
> 
> Maxim Khutornenko wrote:
> This is a matter of consistency with `ThriftBackfill` and while not 
> breaking functionality, could be confusing to have something like 'preferred 
> & prod=false' after a rollback.
> 
> The 'revocable & prod=true' was the invalid combination in the first 
> place, so having rollback fixing consistency here feels right.

So, keep it as is?


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48559/#review137763
---


On June 14, 2016, 10:12 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48559/
> ---
> 
> (Updated June 14, 2016, 10:12 a.m.)
> 
> 
> Review request for Aurora and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - Backfill portion
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 
> f4f8d0037751c9c2096747264c19f6292461b308 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
> 65f5edce74077f52e98f110fcd17b2f12d673f81 
>   src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
> 6fd9ee21cf8e0e42e73a68cdf2d231d581278aae 
>   src/jmh/java/org/apache/aurora/benchmark/Tasks.java 
> e548a09d94b1c6d550f9beec2b7120b64d576f20 
>   src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
> 7497fb96a578cff64b6484c587e29ec464969e84 
>   src/main/java/org/apache/aurora/scheduler/TierManager.java 
> af54cab73a80a5120b1a77fd985dfbaf568d786c 
>   src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> 3ea0992eb0a9930a4db9eb4b7fcab82689495c1f 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
>  0e9562020c298e685e6c2efd18933818b03a5000 
>   
> src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java
>  d08873c88f159eb65b582840b48b7ff604862c31 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V006_PopulateTierField.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java 
> c981a05e61cb053a05144c702c9ffafeb0af8260 
>   
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java 
> 8eed1fc680b0c4fb27d8a353b7f804ae09058156 
>   src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java 
> 0a307fe8d8238c23a526d5c3ee500e1de0761703 
>   src/test/java/org/apache/aurora/codec/ThriftBinaryCodecTest.java 
> 999ef064737c3d8a3d7610b40c13736f51742edd 
>   src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 
> 05cd78f4c7c7d8dd6eeb6f2f9a3e8f7a167f274d 
>   src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
> d4b71f8dbb674384ccbbd9e76f510d127e480e32 
>   src/test/java/org/apache/aurora/scheduler/TierModuleTest.java 
> 58d95dcdf31bc920ca1f8822baccc6c37b66e739 
>   src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 
> 8c8c793813e84bf7ef741f9b6b4ae1e759be1b82 
>   src/test/java/org/apache/aurora/scheduler/base/JobsTest.java 
> 13f656f241a8a9a3d339f4053f165070c2669ef3 
>   src/test/java/org/apache/aurora/scheduler/base/TasksTest.java 
> 935622bef38cc3d399e8b5b1db84fa21c79c78e1 
>   
> src/test/java/org/apache/aurora/scheduler/conf

Re: Review Request 48559: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - Backfill portion

2016-06-17 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48559/
---

(Updated June 17, 2016, 12:46 p.m.)


Review request for Aurora and Maxim Khutornenko.


Changes
---

- Moved ```TaskTestUtil``` back to ```org.apache.aurora.scheduler.base``` 
package
- Moved back fill related logic from ```TierManagerImpl``` to 
```ThriftBackfill```
- Other suggested refactorings

Note: This change set still does not include client changes.


Repository: aurora


Description
---

AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
Job configuration - Backfill portion


Diffs (updated)
-

  src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
65f5edce74077f52e98f110fcd17b2f12d673f81 
  src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
6fd9ee21cf8e0e42e73a68cdf2d231d581278aae 
  src/main/java/org/apache/aurora/scheduler/TierManager.java 
af54cab73a80a5120b1a77fd985dfbaf568d786c 
  src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
3ea0992eb0a9930a4db9eb4b7fcab82689495c1f 
  
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
 0e9562020c298e685e6c2efd18933818b03a5000 
  
src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java 
d08873c88f159eb65b582840b48b7ff604862c31 
  
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V006_PopulateTierField.java
 PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java 
c981a05e61cb053a05144c702c9ffafeb0af8260 
  src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java 
8eed1fc680b0c4fb27d8a353b7f804ae09058156 
  src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java 
0a307fe8d8238c23a526d5c3ee500e1de0761703 
  src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
d4b71f8dbb674384ccbbd9e76f510d127e480e32 
  src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 
8c8c793813e84bf7ef741f9b6b4ae1e759be1b82 
  
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
 2e322d217fc9dc75c51b57607a5547745206fb9f 
  src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java 
e870087e3d47906559410ff76515457f4ff99ff5 
  src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java 
be1132b439948104458efdc82a6bbee43c20c4fd 
  
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java 
e0cf602ead1530301b09eff60287b8fa48be63e8 
  src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java 
0a2cd3d5b01c389f99fca169227aac35436d474b 
  src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java 
4f8158546f3eba8f79d653ad7a30f83d66cbce83 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
6f355d6e66c05651fa9b13356dd81b45bee52adc 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 ecdc62ae3b21b73b6a6af80bb9855867a7e965e0 
  src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 
a54d169caebfc211035386f64169ecd983e378d9 

Diff: https://reviews.apache.org/r/48559/diff/


Testing
---

Manual under Vagrant:
- Deployed old scheduler (with tier backfill support), created a job without 
tier, upgraded scheduler, noticed that tier has been backfilled
- Tried the above scenario with both -use_beta_db_task_store=true and 
-use_beta_db_task_store=false configuration flags
- Verified that if tier is already set it would not be altered
- Verified that it works both when production = 'true' and production = 'false'

End to End:
```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

...

*** OK (All tests passed) ***

mesos-master start/running, process 26886
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real17m53.514s
user0m1.443s
sys 0m0.624s
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 48559: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - Backfill portion

2016-06-20 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48559/
---

(Updated June 20, 2016, 9:47 a.m.)


Review request for Aurora and Maxim Khutornenko.


Changes
---

Applied suggested refactorings


Repository: aurora


Description
---

AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
Job configuration - Backfill portion


Diffs (updated)
-

  src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
65f5edce74077f52e98f110fcd17b2f12d673f81 
  src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 
6fd9ee21cf8e0e42e73a68cdf2d231d581278aae 
  src/main/java/org/apache/aurora/scheduler/TierManager.java 
af54cab73a80a5120b1a77fd985dfbaf568d786c 
  src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
3ea0992eb0a9930a4db9eb4b7fcab82689495c1f 
  
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
 0e9562020c298e685e6c2efd18933818b03a5000 
  
src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java 
d08873c88f159eb65b582840b48b7ff604862c31 
  
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V006_PopulateTierField.java
 PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java 
c981a05e61cb053a05144c702c9ffafeb0af8260 
  src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java 
8eed1fc680b0c4fb27d8a353b7f804ae09058156 
  src/main/java/org/apache/aurora/scheduler/storage/log/ThriftBackfill.java 
0a307fe8d8238c23a526d5c3ee500e1de0761703 
  src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
d4b71f8dbb674384ccbbd9e76f510d127e480e32 
  src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 
8c8c793813e84bf7ef741f9b6b4ae1e759be1b82 
  
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
 2e322d217fc9dc75c51b57607a5547745206fb9f 
  src/test/java/org/apache/aurora/scheduler/storage/backup/RecoveryTest.java 
e870087e3d47906559410ff76515457f4ff99ff5 
  src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java 
be1132b439948104458efdc82a6bbee43c20c4fd 
  
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java 
e0cf602ead1530301b09eff60287b8fa48be63e8 
  src/test/java/org/apache/aurora/scheduler/storage/log/ThriftBackfillTest.java 
0a2cd3d5b01c389f99fca169227aac35436d474b 
  src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java 
4f8158546f3eba8f79d653ad7a30f83d66cbce83 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
6f355d6e66c05651fa9b13356dd81b45bee52adc 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 ecdc62ae3b21b73b6a6af80bb9855867a7e965e0 
  src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 
a54d169caebfc211035386f64169ecd983e378d9 

Diff: https://reviews.apache.org/r/48559/diff/


Testing
---

Manual under Vagrant:
- Deployed old scheduler (with tier backfill support), created a job without 
tier, upgraded scheduler, noticed that tier has been backfilled
- Tried the above scenario with both -use_beta_db_task_store=true and 
-use_beta_db_task_store=false configuration flags
- Verified that if tier is already set it would not be altered
- Verified that it works both when production = 'true' and production = 'false'

End to End:
```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

...

*** OK (All tests passed) ***

mesos-master start/running, process 26886
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real17m53.514s
user0m1.443s
sys 0m0.624s
```


Thanks,

Mehrdad Nurolahzade

Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-21 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/
---

Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Repository: aurora


Description
---

AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
Job configuration - CLI changes


Diffs
-

  src/main/python/apache/aurora/client/api/__init__.py 
68baf8fdb90cd26100159401c46c9963c24332b3 
  src/main/python/apache/aurora/config/__init__.py 
65923be1cb8b88139b8eab0ac5b75428972d3cb1 
  src/main/python/apache/aurora/config/thrift.py 
3539469d243638c0acd08bf0859d0ce858d8977c 
  src/test/python/apache/aurora/client/cli/test_command_hooks.py 
2130f1fa71be02a004cdf8e476a270c81a7105d3 
  src/test/python/apache/aurora/client/cli/test_create.py 
8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
  src/test/python/apache/aurora/client/cli/test_cron.py 
f3c522ed94a2d774865811ceb546bf9df083c14f 
  src/test/python/apache/aurora/client/cli/test_inspect.py 
fedc16b3d4e9fb7d6f5f0dc34ad7a1837e34baea 
  src/test/python/apache/aurora/client/cli/test_plugins.py 
a545fece5e2b3e0017a61e1be9ac478372b1f34d 
  src/test/python/apache/aurora/client/cli/test_restart.py 
967d560e5c7eb0ed85b215fb11d9751b8666acb5 
  src/test/python/apache/aurora/client/cli/util.py 
7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
  src/test/python/apache/aurora/client/test_config.py 
b1a3c1865819899ef19173be0f861783a2631d0a 
  src/test/python/apache/aurora/config/__init__.py PRE-CREATION 
  src/test/python/apache/aurora/config/test_base.py 
b354f0804ce70682e8ecb9fb3a3d8fe736fd1cc5 
  src/test/python/apache/aurora/config/test_thrift.py 
e213184739167e01f3614c20a809af39b3a6b3d6 

Diff: https://reviews.apache.org/r/49048/diff/


Testing
---

Solution fails end to end test, this is WIP intended to receive feedback

I seem to have introduced a dependency between thermos executor and client api 
that is causing the executor fail with the following sample error log:
```
cat 
/var/lib/mesos/slaves/c8fd5700-d2ad-4249-b705-94d40451681b-S0/frameworks/c8fd5700-d2ad-4249-b705-94d40451681b-0001/executors/thermos-www-data-prod-hello-0-f4543715-21f2-402c-9a75-656cb90693b8/runs/19621370-6b72-4c9e-8569-d11c6ca67456/stderr
 
I0621 21:29:00.888162 23895 fetcher.cpp:424] Fetcher Info: 
{"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/c8fd5700-d2ad-4249-b705-94d40451681b-S0\/root","items":[{"action":"BYPASS_CACHE","uri":{"executable":true,"extract":true,"value":"\/home\/vagrant\/aurora\/dist\/thermos_executor.pex"}}],"sandbox_directory":"\/var\/lib\/mesos\/slaves\/c8fd5700-d2ad-4249-b705-94d40451681b-S0\/frameworks\/c8fd5700-d2ad-4249-b705-94d40451681b-0001\/executors\/thermos-www-data-prod-hello-0-f4543715-21f2-402c-9a75-656cb90693b8\/runs\/19621370-6b72-4c9e-8569-d11c6ca67456","user":"root"}
I0621 21:29:00.889114 23895 fetcher.cpp:379] Fetching URI 
'/home/vagrant/aurora/dist/thermos_executor.pex'
I0621 21:29:00.889127 23895 fetcher.cpp:250] Fetching directly into the sandbox 
directory
I0621 21:29:00.889137 23895 fetcher.cpp:187] Fetching URI 
'/home/vagrant/aurora/dist/thermos_executor.pex'
I0621 21:29:00.889147 23895 fetcher.cpp:167] Copying resource with command:cp 
'/home/vagrant/aurora/dist/thermos_executor.pex' 
'/var/lib/mesos/slaves/c8fd5700-d2ad-4249-b705-94d40451681b-S0/frameworks/c8fd5700-d2ad-4249-b705-94d40451681b-0001/executors/thermos-www-data-prod-hello-0-f4543715-21f2-402c-9a75-656cb90693b8/runs/19621370-6b72-4c9e-8569-d11c6ca67456/thermos_executor.pex'
I0621 21:29:00.913426 23895 fetcher.cpp:456] Fetched 
'/home/vagrant/aurora/dist/thermos_executor.pex' to 
'/var/lib/mesos/slaves/c8fd5700-d2ad-4249-b705-94d40451681b-S0/frameworks/c8fd5700-d2ad-4249-b705-94d40451681b-0001/executors/thermos-www-data-prod-hello-0-f4543715-21f2-402c-9a75-656cb90693b8/runs/19621370-6b72-4c9e-8569-d11c6ca67456/thermos_executor.pex'
+ /usr/libexec/mesos/mesos-containerizer mount --help=false 
--operation=make-rslave --path=/
+ grep -E /var/lib/mesos/.+ /proc/self/mountinfo
+ cut -d  -f5
+ xargs --no-run-if-empty umount -l
+ grep -v 19621370-6b72-4c9e-8569-d11c6ca67456
Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 324, in execute
  File ".bootstrap/_pex/pex.py", line 257, in _wrap_coverage
  File ".bootstrap/_pex/pex.py", line 289, in _wrap_profiling
  File ".bootstrap/_pex/pex.py", line 367, in _execute
  File ".bootstrap/_pex/pex.py", line 425, in execute_entry
  File ".bootstrap/_pex/pex.py", line 442, in execute_pkg_resources
  File ".bootstrap/pkg_resources.py", line 2190, in load
  File "apache/aurora/executor/bin/thermos_executor_main.py", line 31, in 

  File "apache/aurora/config/__init__.py", line 28, in 
  File "apache/aurora/config/thrift.py", line 21, in 
ImportError: No module named client.api
```

Not sure if this is the right way to go, please advise.


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-21 Thread Mehrdad Nurolahzade

load
  File "apache/aurora/executor/bin/thermos_executor_main.py", line 31, in 

  File "apache/aurora/config/__init__.py", line 28, in 
  File "apache/aurora/config/thrift.py", line 21, in 
ImportError: No module named client.api
```

Not sure if this is the right way to go, please advise. 
The error obviously goes away by adding 
```src/main/python/apache/aurora/client``` to executor pants BUILD file. But, 
'is that OK if I introduce this dependency?' is what I am asking.


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-21 Thread Mehrdad Nurolahzade



> On June 21, 2016, 4:02 p.m., Maxim Khutornenko wrote:
> > src/main/python/apache/aurora/config/thrift.py, lines 231-258
> > <https://reviews.apache.org/r/49048/diff/1/?file=1427035#file1427035line231>
> >
> > A better place for this would be `get_job_config()` in 
> > apache/aurora/client/cli/context.py. This way a config would always get 
> > populated/backfilled on loading.
> > 
> > Any validaiton/deprecation warnings should go to 
> > apache/aurora/client/config.py.

Yes, you are right shifting to ```context``` causes the thermos executor 
dependency problem to go away, I'll move it.
I initially had deprecation logic in ```config.py``` but then realized thrift 
conversion logic (that causes backfill) happens before this validation so 
pushed it up hierarchy. Now, I can move back to ```config.py``` again.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review138964
-----------


On June 21, 2016, 4:09 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated June 21, 2016, 4:09 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/config/__init__.py 
> 65923be1cb8b88139b8eab0ac5b75428972d3cb1 
>   src/main/python/apache/aurora/config/thrift.py 
> 3539469d243638c0acd08bf0859d0ce858d8977c 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_inspect.py 
> fedc16b3d4e9fb7d6f5f0dc34ad7a1837e34baea 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
>   src/test/python/apache/aurora/config/__init__.py PRE-CREATION 
>   src/test/python/apache/aurora/config/test_base.py 
> b354f0804ce70682e8ecb9fb3a3d8fe736fd1cc5 
>   src/test/python/apache/aurora/config/test_thrift.py 
> e213184739167e01f3614c20a809af39b3a6b3d6 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> Solution fails end to end test, this is WIP intended to receive feedback
> 
> I seem to have introduced a dependency between thermos executor and client 
> api that is causing the executor fail with the following sample error log:
> ```
> cat 
> /var/lib/mesos/slaves/c8fd5700-d2ad-4249-b705-94d40451681b-S0/frameworks/c8fd5700-d2ad-4249-b705-94d40451681b-0001/executors/thermos-www-data-prod-hello-0-f4543715-21f2-402c-9a75-656cb90693b8/runs/19621370-6b72-4c9e-8569-d11c6ca67456/stderr
>  
> I0621 21:29:00.888162 23895 fetcher.cpp:424] Fetcher Info: 
> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/c8fd5700-d2ad-4249-b705-94d40451681b-S0\/root","items":[{"action":"BYPASS_CACHE","uri":{"executable":true,"extract":true,"value":"\/home\/vagrant\/aurora\/dist\/thermos_executor.pex"}}],"sandbox_directory":"\/var\/lib\/mesos\/slaves\/c8fd5700-d2ad-4249-b705-94d40451681b-S0\/frameworks\/c8fd5700-d2ad-4249-b705-94d40451681b-0001\/executors\/thermos-www-data-prod-hello-0-f4543715-21f2-402c-9a75-656cb90693b8\/runs\/19621370-6b72-4c9e-8569-d11c6ca67456","user":"root"}
> I0621 21:29:00.889114 23895 fetcher.cpp:379] Fetching URI 
> '/home/vagrant/aurora/dist/thermos_executor.pex'
> I0621 21:29:00.889127 23895 fetcher.cpp:250] Fetching directly into the 
> sandbox directory
> I0621 21:29:00.889137 23895 fetcher.cpp:187] Fetching URI 
> '/home/vagrant/aurora/dist/

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-23 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/
---

(Updated June 23, 2016, 3:35 p.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Changes
---

- Refactor/Moved backfill logic from ```thrift.py``` to ```context.py```
- Refactor/Moved deprecation warning logic to ```config.py```


Repository: aurora


Description
---

AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
Job configuration - CLI changes


Diffs (updated)
-

  src/main/python/apache/aurora/client/api/__init__.py 
68baf8fdb90cd26100159401c46c9963c24332b3 
  src/main/python/apache/aurora/client/cli/context.py 
9b1511801d031ff48b81c25688a55cb586b8ac66 
  src/main/python/apache/aurora/client/config.py 
2fc12559016d406c347adb416a5166cca31c961e 
  src/main/python/apache/aurora/config/__init__.py 
65923be1cb8b88139b8eab0ac5b75428972d3cb1 
  src/test/python/apache/aurora/client/cli/test_command_hooks.py 
2130f1fa71be02a004cdf8e476a270c81a7105d3 
  src/test/python/apache/aurora/client/cli/test_context.py 
204ca092adad8bf43c5032a02f61bf303fb0b2fc 
  src/test/python/apache/aurora/client/cli/test_create.py 
8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
  src/test/python/apache/aurora/client/cli/test_cron.py 
f3c522ed94a2d774865811ceb546bf9df083c14f 
  src/test/python/apache/aurora/client/cli/test_plugins.py 
a545fece5e2b3e0017a61e1be9ac478372b1f34d 
  src/test/python/apache/aurora/client/cli/test_restart.py 
967d560e5c7eb0ed85b215fb11d9751b8666acb5 
  src/test/python/apache/aurora/client/cli/util.py 
7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
  src/test/python/apache/aurora/client/test_config.py 
b1a3c1865819899ef19173be0f861783a2631d0a 

Diff: https://reviews.apache.org/r/49048/diff/


Testing (updated)
---

```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

*** OK (All tests passed) ***

mesos-master start/running, process 26868
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real19m46.324s
user0m1.496s
sys 0m0.774s
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-23 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review139304
---




src/main/python/apache/aurora/config/__init__.py (line 15)
<https://reviews.apache.org/r/49048/#comment204470>

This is left over from refactoring, will remove.


- Mehrdad Nurolahzade


On June 23, 2016, 3:35 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated June 23, 2016, 3:35 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/main/python/apache/aurora/config/__init__.py 
> 65923be1cb8b88139b8eab0ac5b75428972d3cb1 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-27 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/
---

(Updated June 27, 2016, 9:02 a.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Changes
---

Applied suggested refactorings


Repository: aurora


Description
---

AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
Job configuration - CLI changes


Diffs (updated)
-

  src/main/python/apache/aurora/client/api/__init__.py 
68baf8fdb90cd26100159401c46c9963c24332b3 
  src/main/python/apache/aurora/client/cli/context.py 
9b1511801d031ff48b81c25688a55cb586b8ac66 
  src/main/python/apache/aurora/client/config.py 
2fc12559016d406c347adb416a5166cca31c961e 
  src/test/python/apache/aurora/client/cli/test_command_hooks.py 
2130f1fa71be02a004cdf8e476a270c81a7105d3 
  src/test/python/apache/aurora/client/cli/test_context.py 
204ca092adad8bf43c5032a02f61bf303fb0b2fc 
  src/test/python/apache/aurora/client/cli/test_create.py 
8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
  src/test/python/apache/aurora/client/cli/test_cron.py 
f3c522ed94a2d774865811ceb546bf9df083c14f 
  src/test/python/apache/aurora/client/cli/test_plugins.py 
a545fece5e2b3e0017a61e1be9ac478372b1f34d 
  src/test/python/apache/aurora/client/cli/test_restart.py 
967d560e5c7eb0ed85b215fb11d9751b8666acb5 
  src/test/python/apache/aurora/client/cli/util.py 
7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
  src/test/python/apache/aurora/client/test_config.py 
b1a3c1865819899ef19173be0f861783a2631d0a 

Diff: https://reviews.apache.org/r/49048/diff/


Testing
---

```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

*** OK (All tests passed) ***

mesos-master start/running, process 26868
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real19m46.324s
user0m1.496s
sys 0m0.774s
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-27 Thread Mehrdad Nurolahzade



> On June 27, 2016, 9:22 a.m., Aurora ReviewBot wrote:
> > Master (73dd2a8) is red with this patch.
> >   ./build-support/jenkins/build.sh
> > 
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  ERROR] Could not quitquitquit runner: Cannot take 
> > control of a task in terminal state.
> >  E0627 16:21:42.444761 9324 thermos_task_runner.py:234] 
> > Could not quitquitquit runner: Cannot take control of a task in terminal 
> > state.
> >   generated xml file: 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/dist/test-results/415337499eb72578eab327a6487c1f5c9452b3d6.xml
> >  
> >  [1m[31m 1 failed, 667 passed, 6 skipped, 1 warnings 
> > in 242.05 seconds [0m
> >  
> > FAILURE
> > 
> > [32m
> >Waiting for background workers to finish.[0m
> > 16:22:36 04:46   [complete][31m
> >FAILURE[0m
> > 
> > 
> > I will refresh this build result if you post a review containing 
> > "@ReviewBot retry"

@ReviewBot retry


- Mehrdad


---
This is an

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-27 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review139596
---



@ReviewBot retry

- Mehrdad Nurolahzade


On June 27, 2016, 9:02 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated June 27, 2016, 9:02 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-27 Thread Mehrdad Nurolahzade



> On June 27, 2016, 11:31 a.m., Maxim Khutornenko wrote:
> > src/main/python/apache/aurora/client/cli/context.py, lines 129-130
> > <https://reviews.apache.org/r/49048/diff/3/?file=1431153#file1431153line129>
> >
> > Can you simplify this as:
> > ```
> > return not to_bool(tier.settings['preemptible']) and not 
> > to_bool(tier.settings['revocable'])
> > ```
> > 
> > and falback to `tier_configurations.defaultTierName` in the next()?
> > 
> > Also, I'd change the function name to something less generic, like 
> > `production_tier_filter`.

Do you mean something like the following? (this is breaking my tests)
```
def production_tier_filter(tier):
  return not to_bool(tier.settings['preemptible']) and not 
to_bool(tier.settings['revocable'])

task = config.job().taskConfig
if task.tier is None:
  backfill_args = {
'tier': String(
  next(
(t.name for t in tier_configurations.tiers if 
production_tier_filter(t)),
tier_configurations.defaultTierName))
  }
else:
  backfill_args = {
'production': Boolean(
  next(
(not to_bool(t.settings['preemptible']) for t in 
tier_configurations.tiers if
  production_tier_filter(t)),
task.production))
  }
```


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review139597
---


On June 27, 2016, 9:02 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated June 27, 2016, 9:02 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-27 Thread Mehrdad Nurolahzade



> On June 27, 2016, 11:31 a.m., Maxim Khutornenko wrote:
> > src/main/python/apache/aurora/client/config.py, line 118
> > <https://reviews.apache.org/r/49048/diff/3/?file=1431154#file1431154line118>
> >
> > Please, add a link to our docs for more info: 
> > http://aurora.apache.org/documentation/latest/reference/configuration/#job-objects
> > 
> > Also, release notes need to be updated to clearly state this 
> > deprecation route.

We added an entry to ```RELEASE-NOTES.md``` for 0.14.0 under "Deprecations and 
removals" with a previous JIRA, do we need anything beyond that?
```
- Deprecated `production` field in `TaskConfig` thrift struct. Use `tier` field 
to specify task
  scheduling and resource handling behavior.
```


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review139597
-------


On June 27, 2016, 9:02 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated June 27, 2016, 9:02 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-27 Thread Mehrdad Nurolahzade



> On June 27, 2016, 11:31 a.m., Maxim Khutornenko wrote:
> > src/main/python/apache/aurora/client/cli/context.py, lines 129-130
> > <https://reviews.apache.org/r/49048/diff/3/?file=1431153#file1431153line129>
> >
> > Can you simplify this as:
> > ```
> > return not to_bool(tier.settings['preemptible']) and not 
> > to_bool(tier.settings['revocable'])
> > ```
> > 
> > and falback to `tier_configurations.defaultTierName` in the next()?
> > 
> > Also, I'd change the function name to something less generic, like 
> > `production_tier_filter`.
> 
> Mehrdad Nurolahzade wrote:
> Do you mean something like the following? (this is breaking my tests)
> ```
> def production_tier_filter(tier):
>   return not to_bool(tier.settings['preemptible']) and not 
> to_bool(tier.settings['revocable'])
> 
> task = config.job().taskConfig
> if task.tier is None:
>   backfill_args = {
> 'tier': String(
>   next(
> (t.name for t in tier_configurations.tiers if 
> production_tier_filter(t)),
> tier_configurations.defaultTierName))
>   }
> else:
>   backfill_args = {
> 'production': Boolean(
>   next(
> (not to_bool(t.settings['preemptible']) for t in 
> tier_configurations.tiers if
>   production_tier_filter(t)),
> task.production))
>   }
> ```
> 
> Maxim Khutornenko wrote:
> I presume this is due to the "else" block not accounting for the actual 
> `task.tier` value?

Yes, and the if block also relies on ```task.production``` value. 
I am not sure if I am following your suggested simplification of 
```production_tier_filter``` and using it in both if and else blocks:
```
def production_tier_filter(tier):
  return not to_bool(tier.settings['preemptible']) and not 
to_bool(tier.settings['revocable'])
```


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review139597
---


On June 27, 2016, 9:02 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated June 27, 2016, 9:02 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-28 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/
---

(Updated June 28, 2016, 9:51 a.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Changes
---

- Applied suggested refactorings
- Updated release notes


Repository: aurora


Description
---

AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
Job configuration - CLI changes


Diffs (updated)
-

  RELEASE-NOTES.md af2061c7605c12a066778bd99ec1a3857bee6dec 
  src/main/python/apache/aurora/client/api/__init__.py 
68baf8fdb90cd26100159401c46c9963c24332b3 
  src/main/python/apache/aurora/client/base.py 
73953b69b247cf86e691310423b63b3a40f95976 
  src/main/python/apache/aurora/client/cli/context.py 
9b1511801d031ff48b81c25688a55cb586b8ac66 
  src/main/python/apache/aurora/client/config.py 
2fc12559016d406c347adb416a5166cca31c961e 
  src/test/python/apache/aurora/client/cli/test_command_hooks.py 
2130f1fa71be02a004cdf8e476a270c81a7105d3 
  src/test/python/apache/aurora/client/cli/test_context.py 
204ca092adad8bf43c5032a02f61bf303fb0b2fc 
  src/test/python/apache/aurora/client/cli/test_create.py 
8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
  src/test/python/apache/aurora/client/cli/test_cron.py 
f3c522ed94a2d774865811ceb546bf9df083c14f 
  src/test/python/apache/aurora/client/cli/test_plugins.py 
a545fece5e2b3e0017a61e1be9ac478372b1f34d 
  src/test/python/apache/aurora/client/cli/test_restart.py 
967d560e5c7eb0ed85b215fb11d9751b8666acb5 
  src/test/python/apache/aurora/client/cli/util.py 
7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
  src/test/python/apache/aurora/client/test_config.py 
b1a3c1865819899ef19173be0f861783a2631d0a 

Diff: https://reviews.apache.org/r/49048/diff/


Testing
---

```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

*** OK (All tests passed) ***

mesos-master start/running, process 26868
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real19m46.324s
user0m1.496s
sys 0m0.774s
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-28 Thread Mehrdad Nurolahzade



> On June 28, 2016, 10:02 a.m., Aurora ReviewBot wrote:
> > Master (73dd2a8) is red with this patch.
> >   ./build-support/jenkins/build.sh
> > 
> >  
> > src/test/python/apache/thermos/common/test_planner.py::test_planner_empty 
> > [32mPASSED[0m
> >  
> > src/test/python/apache/thermos/common/test_planner.py::test_planner_unordered
> >  [32mPASSED[0m
> >  
> > src/test/python/apache/thermos/common/test_planner.py::test_planner_ordered 
> > [32mPASSED[0m
> >  
> > src/test/python/apache/thermos/common/test_planner.py::test_planner_mixed 
> > [32mPASSED[0m
> >  
> > src/test/python/apache/thermos/common/test_planner.py::test_planner_unsatisfiables
> >  [32mPASSED[0m
> >  
> >   FAILURES 
> >   
> > test_get_config_with_production_and_tier_is_preemptible 
> >  
> >  [1mdef 
> > test_get_config_with_production_and_tier_is_preemptible():[0m
> >  [1m  context = FakeAuroraCommandContext()[0m
> >  [1m  
> > context.set_options(create_mock_options())[0m
> >  [1m  with 
> > CLUSTERS.patch(AuroraClientCommandTest.TEST_CLUSTERS.values()):[0m
> >  [1mapi = 
> > context.get_api(TEST_CLUSTER.name)[0m
> >  [1mapi.get_tier_configs.return_value = 
> > AuroraClientCommandTest.get_mock_tier_configurations()[0m
> >  [1mwith temporary_file() as fp:[0m
> >  [1m  fp.write(create_test_config())[0m
> >  [1m  fp.flush()[0m
> >  [1m  config = 
> > context.get_job_config(AuroraClientCommandTest.TEST_JOBKEY, fp.name)[0m
> >  [1m  assert not 
> > config.job().taskConfig.production[0m
> >  [1m> assert config.job().taskConfig.tier == 
> > AuroraClientCommandTest.PREEMPTIBLE_TIER.name[0m
> >  [1m[31mE assert 'revocable' == 
> > 'preemptible'[0m
> >  [1m[31mE   - revocable[0m
> >  [1m[31mE   + preemptible[0m
> >  
> >  
> > src/test/python/apache/aurora/client/cli/test_context.py:110: AssertionError
> >  -- Captured stderr call --
> >  INFO:root:OK
> >   INFO] OK
> >   INFO] OK
> >   INFO] OK
> >   generated xml file: 
> > /home/jenkins/jenkins-slave/workspace/AuroraBot/dist/test-results/415337499eb72578eab327a6487c1f5c9452b3d6.xml
> >  
> >  [1m[31m 1 failed, 667 passed, 6 skipped, 1 warnings 
> > in 198.50 seconds [0m
> >  
> > FAILURE
> > 
> > [32m
> >Waiting for background workers to finish.[0m
> > 17:01:40 04:08   [complete][31m
> >FAILURE[0m
> > 
> > 
> > I will refresh this build result if you post a review containing 
> > "@ReviewBot retry"

This is passing locally, not sure why it's breaking on the server.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review139819
---


On June 28, 2016, 10:28 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated June 28, 2016, 10:28 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md af2061c7605c12a066778bd99ec1a3857bee6dec 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
>

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-28 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/
---

(Updated June 28, 2016, 10:28 a.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Changes
---

Applied suggested refactoring


Repository: aurora


Description
---

AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
Job configuration - CLI changes


Diffs (updated)
-

  RELEASE-NOTES.md af2061c7605c12a066778bd99ec1a3857bee6dec 
  src/main/python/apache/aurora/client/api/__init__.py 
68baf8fdb90cd26100159401c46c9963c24332b3 
  src/main/python/apache/aurora/client/cli/context.py 
9b1511801d031ff48b81c25688a55cb586b8ac66 
  src/main/python/apache/aurora/client/config.py 
2fc12559016d406c347adb416a5166cca31c961e 
  src/test/python/apache/aurora/client/cli/test_command_hooks.py 
2130f1fa71be02a004cdf8e476a270c81a7105d3 
  src/test/python/apache/aurora/client/cli/test_context.py 
204ca092adad8bf43c5032a02f61bf303fb0b2fc 
  src/test/python/apache/aurora/client/cli/test_create.py 
8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
  src/test/python/apache/aurora/client/cli/test_cron.py 
f3c522ed94a2d774865811ceb546bf9df083c14f 
  src/test/python/apache/aurora/client/cli/test_plugins.py 
a545fece5e2b3e0017a61e1be9ac478372b1f34d 
  src/test/python/apache/aurora/client/cli/test_restart.py 
967d560e5c7eb0ed85b215fb11d9751b8666acb5 
  src/test/python/apache/aurora/client/cli/util.py 
7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
  src/test/python/apache/aurora/client/test_config.py 
b1a3c1865819899ef19173be0f861783a2631d0a 

Diff: https://reviews.apache.org/r/49048/diff/


Testing
---

```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

*** OK (All tests passed) ***

mesos-master start/running, process 26868
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real19m46.324s
user0m1.496s
sys 0m0.774s
```


Thanks,

Mehrdad Nurolahzade

Review Request 49334: AURORA-1725 Expose tier configurations as a debug page

2016-06-28 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49334/
---

Review request for Aurora, David McLaughlin and Maxim Khutornenko.


Repository: aurora


Description
---

AURORA-1725 Expose tier configurations as a debug page


Diffs
-

  config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
  src/main/java/org/apache/aurora/scheduler/http/JettyServerModule.java 
496cd5590d6042d30e7bd0410a6c40d804c15023 
  src/main/java/org/apache/aurora/scheduler/http/Tiers.java PRE-CREATION 
  src/main/resources/scheduler/assets/index.html 
eca27e081950a66b743f79ae7ae3d08e4625d2c8 
  src/test/java/org/apache/aurora/scheduler/http/AbstractJettyTest.java 
561b134152113a50ff495cef28d2396b56d02f1c 

Diff: https://reviews.apache.org/r/49334/diff/


Testing
---

Comments:
- Logged the work done so far (new ```/tiers``` debug page) to this ticket and 
kept AURORA-1710 for modifications to scheduler UI. 
- Added comments to AURORA-1710 to clarify the suggested approach for 
implementation.

Testing:
```
./build-support/jenkins/build.sh
...

*** OK (All tests passed) ***

./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
...
mesos-master start/running, process 18776
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real15m5.795s
user0m1.318s
sys 0m0.533s
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49334: AURORA-1725 Expose tier configurations as a debug page

2016-06-28 Thread Mehrdad Nurolahzade

> On June 28, 2016, 1:53 p.m., David McLaughlin wrote:
> > config/legacy_untested_classes.txt, line 44
> > <https://reviews.apache.org/r/49334/diff/1/?file=1432053#file1432053line44>
> >
> > Isn't this a fairly easy class to test?

I wrote a test for it initially but then I ran into a few problems interesting 
problems but then eventually gave up. Seems like many other http endpoints have 
remained untested for a similar reason. I can get behind finding a solution for 
this testing problem though.

The ```getTiers()``` call uses ```MessageBodyWriter``` to serialize the 
```Map```. The elements in that map can be ordered randomly (depending on map 
implementation) which makes String equality difficult. That was the easier 
problem to solve, I used ```LinkedHashMap``` in test fixtures to force 
determinstic order of elements in the map.

Then, I ran into the problem of Jackson ordering ```TierInfo``` properties in 
undeterminstic order. That is, a tier can be serialized as 
```"perferred":{"preemptible":false,"revocable":false}``` or 
```"perferred":{"revocable":false,"preemptible":false}```. A quick Google 
search suggested that application of @JasonPropertyOrder on ```TierInfo``` can 
result in deterministic property serialization order but it did not work for me 
(potentially because of mocking but I am not sure).

Anyways, after burning some time here I decided to give up. Let me know if you 
have any suggestion for testing.

- Mehrdad

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49334/#review139857
-------

On June 28, 2016, 1:42 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49334/
> ---
> 
> (Updated June 28, 2016, 1:42 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1725 Expose tier configurations as a debug page
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
>   src/main/java/org/apache/aurora/scheduler/http/JettyServerModule.java 
> 496cd5590d6042d30e7bd0410a6c40d804c15023 
>   src/main/java/org/apache/aurora/scheduler/http/Tiers.java PRE-CREATION 
>   src/main/resources/scheduler/assets/index.html 
> eca27e081950a66b743f79ae7ae3d08e4625d2c8 
>   src/test/java/org/apache/aurora/scheduler/http/AbstractJettyTest.java 
> 561b134152113a50ff495cef28d2396b56d02f1c 
> 
> Diff: https://reviews.apache.org/r/49334/diff/
> 
> 
> Testing
> ---
> 
> Comments:
> - Logged the work done so far (new ```/tiers``` debug page) to this ticket 
> and kept AURORA-1710 for modifications to scheduler UI. 
> - Added comments to AURORA-1710 to clarify the suggested approach for 
> implementation.
> 
> Testing:
> ```
> ./build-support/jenkins/build.sh
> ...
> 
> *** OK (All tests passed) ***
> 
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ...
> mesos-master start/running, process 18776
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  15m5.795s
> user  0m1.318s
> sys   0m0.533s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49334: AURORA-1725 Expose tier configurations as a debug page

2016-06-28 Thread Mehrdad Nurolahzade



> On June 28, 2016, 1:53 p.m., David McLaughlin wrote:
> > config/legacy_untested_classes.txt, line 44
> > <https://reviews.apache.org/r/49334/diff/1/?file=1432053#file1432053line44>
> >
> > Isn't this a fairly easy class to test?
> 
> Mehrdad Nurolahzade wrote:
> I wrote a test for it initially but then I ran into a few problems 
> interesting problems but then eventually gave up. Seems like many other http 
> endpoints have remained untested for a similar reason. I can get behind 
> finding a solution for this testing problem though.
> 
> The ```getTiers()``` call uses ```MessageBodyWriter``` to serialize the 
> ```Map```. The elements in that map can be ordered randomly (depending on map 
> implementation) which makes String equality difficult. That was the easier 
> problem to solve, I used ```LinkedHashMap``` in test fixtures to force 
> determinstic order of elements in the map.
> 
> Then, I ran into the problem of Jackson ordering ```TierInfo``` 
> properties in undeterminstic order. That is, a tier can be serialized as 
> ```"perferred":{"preemptible":false,"revocable":false}``` or 
> ```"perferred":{"revocable":false,"preemptible":false}```. A quick Google 
> search suggested that application of @JasonPropertyOrder on ```TierInfo``` 
> can result in deterministic property serialization order but it did not work 
> for me (potentially because of mocking but I am not sure).
> 
> Anyways, after burning some time here I decided to give up. Let me know 
> if you have any suggestion for testing.

Found a simple solution using GSON. Will update this review board shortly.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49334/#review139857
---


On June 28, 2016, 1:42 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49334/
> ---
> 
> (Updated June 28, 2016, 1:42 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1725 Expose tier configurations as a debug page
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
>   src/main/java/org/apache/aurora/scheduler/http/JettyServerModule.java 
> 496cd5590d6042d30e7bd0410a6c40d804c15023 
>   src/main/java/org/apache/aurora/scheduler/http/Tiers.java PRE-CREATION 
>   src/main/resources/scheduler/assets/index.html 
> eca27e081950a66b743f79ae7ae3d08e4625d2c8 
>   src/test/java/org/apache/aurora/scheduler/http/AbstractJettyTest.java 
> 561b134152113a50ff495cef28d2396b56d02f1c 
> 
> Diff: https://reviews.apache.org/r/49334/diff/
> 
> 
> Testing
> ---
> 
> Comments:
> - Logged the work done so far (new ```/tiers``` debug page) to this ticket 
> and kept AURORA-1710 for modifications to scheduler UI. 
> - Added comments to AURORA-1710 to clarify the suggested approach for 
> implementation.
> 
> Testing:
> ```
> ./build-support/jenkins/build.sh
> ...
> 
> *** OK (All tests passed) ***
> 
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ...
> mesos-master start/running, process 18776
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  15m5.795s
> user  0m1.318s
> sys   0m0.533s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49334: AURORA-1725 Expose tier configurations as a debug page

2016-06-28 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49334/
---

(Updated June 28, 2016, 3:04 p.m.)


Review request for Aurora, David McLaughlin and Maxim Khutornenko.


Changes
---

Added unit test


Repository: aurora


Description
---

AURORA-1725 Expose tier configurations as a debug page


Diffs (updated)
-

  src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
4089b79da8079243703eead884e80bcf736f8b29 
  src/main/java/org/apache/aurora/scheduler/http/JettyServerModule.java 
496cd5590d6042d30e7bd0410a6c40d804c15023 
  src/main/java/org/apache/aurora/scheduler/http/Tiers.java PRE-CREATION 
  src/main/resources/scheduler/assets/index.html 
eca27e081950a66b743f79ae7ae3d08e4625d2c8 
  src/test/java/org/apache/aurora/scheduler/TierManagerTest.java 
7d9b3330b8c2b7a87b4dc3adfff94e40ef25d294 
  src/test/java/org/apache/aurora/scheduler/http/AbstractJettyTest.java 
561b134152113a50ff495cef28d2396b56d02f1c 
  src/test/java/org/apache/aurora/scheduler/http/TiersTest.java PRE-CREATION 

Diff: https://reviews.apache.org/r/49334/diff/


Testing
---

Comments:
- Logged the work done so far (new ```/tiers``` debug page) to this ticket and 
kept AURORA-1710 for modifications to scheduler UI. 
- Added comments to AURORA-1710 to clarify the suggested approach for 
implementation.

Testing:
```
./build-support/jenkins/build.sh
...

*** OK (All tests passed) ***

./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
...
mesos-master start/running, process 18776
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real15m5.795s
user0m1.318s
sys 0m0.533s
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-06-28 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/
---

(Updated June 28, 2016, 4:57 p.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Bugs: AURORA-1710
https://issues.apache.org/jira/browse/AURORA-1710


Repository: aurora


Description
---

AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
Job configuration - CLI changes


Diffs
-

  RELEASE-NOTES.md af2061c7605c12a066778bd99ec1a3857bee6dec 
  src/main/python/apache/aurora/client/api/__init__.py 
68baf8fdb90cd26100159401c46c9963c24332b3 
  src/main/python/apache/aurora/client/cli/context.py 
9b1511801d031ff48b81c25688a55cb586b8ac66 
  src/main/python/apache/aurora/client/config.py 
2fc12559016d406c347adb416a5166cca31c961e 
  src/test/python/apache/aurora/client/cli/test_command_hooks.py 
2130f1fa71be02a004cdf8e476a270c81a7105d3 
  src/test/python/apache/aurora/client/cli/test_context.py 
204ca092adad8bf43c5032a02f61bf303fb0b2fc 
  src/test/python/apache/aurora/client/cli/test_create.py 
8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
  src/test/python/apache/aurora/client/cli/test_cron.py 
f3c522ed94a2d774865811ceb546bf9df083c14f 
  src/test/python/apache/aurora/client/cli/test_plugins.py 
a545fece5e2b3e0017a61e1be9ac478372b1f34d 
  src/test/python/apache/aurora/client/cli/test_restart.py 
967d560e5c7eb0ed85b215fb11d9751b8666acb5 
  src/test/python/apache/aurora/client/cli/util.py 
7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
  src/test/python/apache/aurora/client/test_config.py 
b1a3c1865819899ef19173be0f861783a2631d0a 

Diff: https://reviews.apache.org/r/49048/diff/


Testing
---

```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

*** OK (All tests passed) ***

mesos-master start/running, process 26868
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real19m46.324s
user0m1.496s
sys 0m0.774s
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-08 Thread Mehrdad Nurolahzade



> On June 28, 2016, 8:39 p.m., Joshua Cohen wrote:
> >

Should this be merged with upstream now that 0.15 is released?


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review139923
---


On June 28, 2016, 4:57 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated June 28, 2016, 4:57 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1710
> https://issues.apache.org/jira/browse/AURORA-1710
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md af2061c7605c12a066778bd99ec1a3857bee6dec 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-08 Thread Mehrdad Nurolahzade



> On June 28, 2016, 8:39 p.m., Joshua Cohen wrote:
> >
> 
> Mehrdad Nurolahzade wrote:
> Should this be merged with upstream now that 0.15 is released?
> 
> Joshua Cohen wrote:
> Yes, we can land this now. Would you mind rebasing? Then I can commit it.

Alright, running tests locally now, going to push shortly.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review139923
---


On June 28, 2016, 4:57 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated June 28, 2016, 4:57 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1710
> https://issues.apache.org/jira/browse/AURORA-1710
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md af2061c7605c12a066778bd99ec1a3857bee6dec 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-08 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/
---

(Updated July 8, 2016, 1:12 p.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Changes
---

Rebased to master


Bugs: AURORA-1710
https://issues.apache.org/jira/browse/AURORA-1710


Repository: aurora


Description
---

AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
Job configuration - CLI changes


Diffs (updated)
-

  RELEASE-NOTES.md 29d224d5596eb060356f1343cf347db79b1ad687 
  src/main/python/apache/aurora/client/api/__init__.py 
68baf8fdb90cd26100159401c46c9963c24332b3 
  src/main/python/apache/aurora/client/cli/context.py 
9b1511801d031ff48b81c25688a55cb586b8ac66 
  src/main/python/apache/aurora/client/config.py 
2fc12559016d406c347adb416a5166cca31c961e 
  src/test/python/apache/aurora/client/cli/test_command_hooks.py 
2130f1fa71be02a004cdf8e476a270c81a7105d3 
  src/test/python/apache/aurora/client/cli/test_context.py 
204ca092adad8bf43c5032a02f61bf303fb0b2fc 
  src/test/python/apache/aurora/client/cli/test_create.py 
8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
  src/test/python/apache/aurora/client/cli/test_cron.py 
f3c522ed94a2d774865811ceb546bf9df083c14f 
  src/test/python/apache/aurora/client/cli/test_plugins.py 
a545fece5e2b3e0017a61e1be9ac478372b1f34d 
  src/test/python/apache/aurora/client/cli/test_restart.py 
967d560e5c7eb0ed85b215fb11d9751b8666acb5 
  src/test/python/apache/aurora/client/cli/util.py 
7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
  src/test/python/apache/aurora/client/test_config.py 
b1a3c1865819899ef19173be0f861783a2631d0a 

Diff: https://reviews.apache.org/r/49048/diff/


Testing
---

```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

*** OK (All tests passed) ***

mesos-master start/running, process 26868
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real19m46.324s
user0m1.496s
sys 0m0.774s
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-08 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review141329
---



@ReviewBot retry

- Mehrdad Nurolahzade


On July 8, 2016, 1:12 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated July 8, 2016, 1:12 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1710
> https://issues.apache.org/jira/browse/AURORA-1710
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 29d224d5596eb060356f1343cf347db79b1ad687 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-08 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/
---

(Updated July 8, 2016, 4:15 p.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Changes
---

The logic was originally correct but then I modified it based on review 
comments without noticing that the refactoring has broken the logic.


Bugs: AURORA-1710
https://issues.apache.org/jira/browse/AURORA-1710


Repository: aurora


Description
---

AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
Job configuration - CLI changes


Diffs (updated)
-

  RELEASE-NOTES.md 29d224d5596eb060356f1343cf347db79b1ad687 
  src/main/python/apache/aurora/client/api/__init__.py 
68baf8fdb90cd26100159401c46c9963c24332b3 
  src/main/python/apache/aurora/client/cli/context.py 
9b1511801d031ff48b81c25688a55cb586b8ac66 
  src/main/python/apache/aurora/client/config.py 
2fc12559016d406c347adb416a5166cca31c961e 
  src/test/python/apache/aurora/client/cli/test_command_hooks.py 
2130f1fa71be02a004cdf8e476a270c81a7105d3 
  src/test/python/apache/aurora/client/cli/test_context.py 
204ca092adad8bf43c5032a02f61bf303fb0b2fc 
  src/test/python/apache/aurora/client/cli/test_create.py 
8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
  src/test/python/apache/aurora/client/cli/test_cron.py 
f3c522ed94a2d774865811ceb546bf9df083c14f 
  src/test/python/apache/aurora/client/cli/test_plugins.py 
a545fece5e2b3e0017a61e1be9ac478372b1f34d 
  src/test/python/apache/aurora/client/cli/test_restart.py 
967d560e5c7eb0ed85b215fb11d9751b8666acb5 
  src/test/python/apache/aurora/client/cli/util.py 
7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
  src/test/python/apache/aurora/client/test_config.py 
b1a3c1865819899ef19173be0f861783a2631d0a 

Diff: https://reviews.apache.org/r/49048/diff/


Testing
---

```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

*** OK (All tests passed) ***

mesos-master start/running, process 26868
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real19m46.324s
user0m1.496s
sys 0m0.774s
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-11 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review141698
---



@ReviewBot retry

- Mehrdad Nurolahzade


On July 8, 2016, 4:15 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated July 8, 2016, 4:15 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1710
> https://issues.apache.org/jira/browse/AURORA-1710
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 29d224d5596eb060356f1343cf347db79b1ad687 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Review Request 50052: AURORA-1736 Display reservations and persistent volumes in /offers debug http endpoint

2016-07-14 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50052/
---

Review request for Aurora, Joshua Cohen and Stephan Erb.


Bugs: AURORA-1736
https://issues.apache.org/jira/browse/AURORA-1736


Repository: aurora


Description
---

AURORA-1736 Display reservations and persistent volumes in /offers debug http 
endpoint


Diffs
-

  src/main/java/org/apache/aurora/scheduler/http/Offers.java 
80f082410896a50d86c7886736caf79581f5051c 

Diff: https://reviews.apache.org/r/50052/diff/


Testing
---

Manual, Jenkins, and end_to_end


Thanks,

Mehrdad Nurolahzade

Re: Review Request 50052: AURORA-1736 Display reservations and persistent volumes in /offers debug http endpoint

2016-07-15 Thread Mehrdad Nurolahzade



> On July 14, 2016, 7:34 p.m., Joshua Cohen wrote:
> > How do you feel about adding a unit test for `getOffers` that verifies all 
> > this new logic?

I feel good. :)


> On July 14, 2016, 7:34 p.m., Joshua Cohen wrote:
> > src/main/java/org/apache/aurora/scheduler/http/Offers.java, line 89
> > <https://reviews.apache.org/r/50052/diff/1/?file=1444281#file1444281line89>
> >
> > Is there any reason that all of these `*_TO_BEAN` functions return an 
> > Object rather than `Map` (since that's what the root `TO_BEAN` 
> > function returns)

Correct, going to refactor all.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50052/#review142326
-----------


On July 14, 2016, 4:08 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50052/
> ---
> 
> (Updated July 14, 2016, 4:08 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Stephan Erb.
> 
> 
> Bugs: AURORA-1736
> https://issues.apache.org/jira/browse/AURORA-1736
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1736 Display reservations and persistent volumes in /offers debug http 
> endpoint
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/http/Offers.java 
> 80f082410896a50d86c7886736caf79581f5051c 
> 
> Diff: https://reviews.apache.org/r/50052/diff/
> 
> 
> Testing
> ---
> 
> Manual, Jenkins, and end_to_end
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50052: AURORA-1736 Display reservations and persistent volumes in /offers debug http endpoint

2016-07-15 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50052/
---

(Updated July 15, 2016, 1:15 p.m.)


Review request for Aurora, Joshua Cohen and Stephan Erb.


Changes
---

- Added unit test
- Applied suggested refactoring


Bugs: AURORA-1736
https://issues.apache.org/jira/browse/AURORA-1736


Repository: aurora


Description
---

AURORA-1736 Display reservations and persistent volumes in /offers debug http 
endpoint


Diffs (updated)
-

  config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
  src/main/java/org/apache/aurora/scheduler/http/Offers.java 
80f082410896a50d86c7886736caf79581f5051c 
  src/test/java/org/apache/aurora/scheduler/http/OffersTest.java PRE-CREATION 

Diff: https://reviews.apache.org/r/50052/diff/


Testing
---

Manual, Jenkins, and end_to_end


Thanks,

Mehrdad Nurolahzade

Re: Review Request 50052: AURORA-1736 Display reservations and persistent volumes in /offers debug http endpoint

2016-07-18 Thread Mehrdad Nurolahzade



> On July 17, 2016, 3:15 p.m., Stephan Erb wrote:
> > Should we consider adopting `protobuf-java-util`'s JsonFormatter rather 
> > than implementing this on our own? 
> > 
> > * 
> > https://github.com/google/protobuf/blob/master/java/util/src/main/java/com/google/protobuf/util/JsonFormat.java
> > * 
> > http://mvnrepository.com/artifact/com.google.protobuf/protobuf-java-util/3.0.0-beta-3

Looking into protobuf util ...


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50052/#review142524
-------


On July 15, 2016, 1:15 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50052/
> ---
> 
> (Updated July 15, 2016, 1:15 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Stephan Erb.
> 
> 
> Bugs: AURORA-1736
> https://issues.apache.org/jira/browse/AURORA-1736
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1736 Display reservations and persistent volumes in /offers debug http 
> endpoint
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
>   src/main/java/org/apache/aurora/scheduler/http/Offers.java 
> 80f082410896a50d86c7886736caf79581f5051c 
>   src/test/java/org/apache/aurora/scheduler/http/OffersTest.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50052/diff/
> 
> 
> Testing
> ---
> 
> Manual, Jenkins, and end_to_end
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50052: AURORA-1736 Display reservations and persistent volumes in /offers debug http endpoint

2016-07-18 Thread Mehrdad Nurolahzade



> On July 17, 2016, 3:15 p.m., Stephan Erb wrote:
> > Should we consider adopting `protobuf-java-util`'s JsonFormatter rather 
> > than implementing this on our own? 
> > 
> > * 
> > https://github.com/google/protobuf/blob/master/java/util/src/main/java/com/google/protobuf/util/JsonFormat.java
> > * 
> > http://mvnrepository.com/artifact/com.google.protobuf/protobuf-java-util/3.0.0-beta-3
> 
> Mehrdad Nurolahzade wrote:
> Looking into protobuf util ...

I can see that we already have 
[jackson-datatype-protobuf](https://github.com/HubSpot/jackson-datatype-protobuf)
 library in the project that can be used to create a JSON serializer from 
protobuf. We can probably get this task done using it. However, 
```protobuf-java-util``` seems to have a larger and more active community.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50052/#review142524
-------


On July 15, 2016, 1:15 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50052/
> ---
> 
> (Updated July 15, 2016, 1:15 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Stephan Erb.
> 
> 
> Bugs: AURORA-1736
> https://issues.apache.org/jira/browse/AURORA-1736
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1736 Display reservations and persistent volumes in /offers debug http 
> endpoint
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
>   src/main/java/org/apache/aurora/scheduler/http/Offers.java 
> 80f082410896a50d86c7886736caf79581f5051c 
>   src/test/java/org/apache/aurora/scheduler/http/OffersTest.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50052/diff/
> 
> 
> Testing
> ---
> 
> Manual, Jenkins, and end_to_end
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50052: AURORA-1736 Display reservations and persistent volumes in /offers debug http endpoint

2016-07-18 Thread Mehrdad Nurolahzade



> On July 17, 2016, 3:15 p.m., Stephan Erb wrote:
> > Should we consider adopting `protobuf-java-util`'s JsonFormatter rather 
> > than implementing this on our own? 
> > 
> > * 
> > https://github.com/google/protobuf/blob/master/java/util/src/main/java/com/google/protobuf/util/JsonFormat.java
> > * 
> > http://mvnrepository.com/artifact/com.google.protobuf/protobuf-java-util/3.0.0-beta-3
> 
> Mehrdad Nurolahzade wrote:
> Looking into protobuf util ...
> 
> Mehrdad Nurolahzade wrote:
> I can see that we already have 
> [jackson-datatype-protobuf](https://github.com/HubSpot/jackson-datatype-protobuf)
>  library in the project that can be used to create a JSON serializer from 
> protobuf. We can probably get this task done using it. However, 
> ```protobuf-java-util``` seems to have a larger and more active community.

I spent some time trying to get ```protobuf-java-util``` to work. However, it 
seems to a very light-weight utility designed to creat one-to-one mapping from 
protobuf to JSON. Controls to exclude or give special treatment to fields in 
the mapping are non-existant which makes the resulting generated JSON look 
significantly different from what we are serving today.

Then I looked into ```jackson-datatype-protobuf``` and I found a great degree 
of flexibility in it to control generated JSON that I did not see in 
```protobuf-java-util```. To generate JSON output:

```
  @GET
  @Produces(MediaType.APPLICATION_JSON)
  public Response getOffers() throws JsonProcessingException {
ObjectMapper mapper = new ObjectMapper()
.registerModule(new ProtobufModule())

.setPropertyNamingStrategy(PropertyNamingStrategy.CAMEL_CASE_TO_LOWER_CASE_WITH_UNDERSCORES)
.setSerializationInclusion(JsonInclude.Include.NON_NULL);
return 
Response.ok(mapper.writeValueAsString(offerManager.getOffers())).build();
  }
```

However, the resulting JSON output is different from what we used to serve (see 
the attachments). If this change is acceptable (and won't break anything else) 
I can use it as-is. Otherwise, I have to write code to customize the output 
(filter out certain fields, change rendering, etc.) which I feel somehow 
defeats the purpose of this refactoring.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50052/#review142524
-----------


On July 15, 2016, 1:15 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50052/
> ---
> 
> (Updated July 15, 2016, 1:15 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Stephan Erb.
> 
> 
> Bugs: AURORA-1736
> https://issues.apache.org/jira/browse/AURORA-1736
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1736 Display reservations and persistent volumes in /offers debug http 
> endpoint
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
>   src/main/java/org/apache/aurora/scheduler/http/Offers.java 
> 80f082410896a50d86c7886736caf79581f5051c 
>   src/test/java/org/apache/aurora/scheduler/http/OffersTest.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50052/diff/
> 
> 
> Testing
> ---
> 
> Manual, Jenkins, and end_to_end
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50052: AURORA-1736 Display reservations and persistent volumes in /offers debug http endpoint

2016-07-18 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50052/
---

(Updated July 18, 2016, 12:32 p.m.)


Review request for Aurora, Joshua Cohen and Stephan Erb.


Bugs: AURORA-1736
https://issues.apache.org/jira/browse/AURORA-1736


Repository: aurora


Description
---

AURORA-1736 Display reservations and persistent volumes in /offers debug http 
endpoint


Diffs
-

  config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
  src/main/java/org/apache/aurora/scheduler/http/Offers.java 
80f082410896a50d86c7886736caf79581f5051c 
  src/test/java/org/apache/aurora/scheduler/http/OffersTest.java PRE-CREATION 

Diff: https://reviews.apache.org/r/50052/diff/


Testing
---

Manual, Jenkins, and end_to_end


File Attachments (updated)


CURRENT
  
https://reviews.apache.org/media/uploaded/files/2016/07/18/1de4c357-c932-4c84-962f-4209a5b679bc__offers-old.json
MODIFIED
  
https://reviews.apache.org/media/uploaded/files/2016/07/18/50faae5c-45ac-45af-aa3c-d68ca09d2e72__offers-new.json


Thanks,

Mehrdad Nurolahzade

Re: Review Request 50052: AURORA-1736 Display reservations and persistent volumes in /offers debug http endpoint

2016-07-18 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50052/
---

(Updated July 18, 2016, 7:11 p.m.)


Review request for Aurora, Joshua Cohen and Stephan Erb.


Changes
---

Rendering JSON using ```jackson-datatype-protobuf``` library


Bugs: AURORA-1736
https://issues.apache.org/jira/browse/AURORA-1736


Repository: aurora


Description
---

AURORA-1736 Display reservations and persistent volumes in /offers debug http 
endpoint


Diffs (updated)
-

  config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
  src/main/java/org/apache/aurora/scheduler/http/Offers.java 
80f082410896a50d86c7886736caf79581f5051c 
  src/test/java/org/apache/aurora/scheduler/http/OffersTest.java PRE-CREATION 

Diff: https://reviews.apache.org/r/50052/diff/


Testing
---

Manual, Jenkins, and end_to_end


File Attachments


CURRENT
  
https://reviews.apache.org/media/uploaded/files/2016/07/18/1de4c357-c932-4c84-962f-4209a5b679bc__offers-old.json
MODIFIED
  
https://reviews.apache.org/media/uploaded/files/2016/07/18/50faae5c-45ac-45af-aa3c-d68ca09d2e72__offers-new.json


Thanks,

Mehrdad Nurolahzade

Re: Review Request 50052: AURORA-1736 Display reservations and persistent volumes in /offers debug http endpoint

2016-07-18 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50052/
---

(Updated July 18, 2016, 7:12 p.m.)


Review request for Aurora, Joshua Cohen and Stephan Erb.


Bugs: AURORA-1736
https://issues.apache.org/jira/browse/AURORA-1736


Repository: aurora


Description
---

AURORA-1736 Display reservations and persistent volumes in /offers debug http 
endpoint


Diffs
-

  config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
  src/main/java/org/apache/aurora/scheduler/http/Offers.java 
80f082410896a50d86c7886736caf79581f5051c 
  src/test/java/org/apache/aurora/scheduler/http/OffersTest.java PRE-CREATION 

Diff: https://reviews.apache.org/r/50052/diff/


Testing
---

Manual, Jenkins, and end_to_end


File Attachments (updated)


CURRENT
  
https://reviews.apache.org/media/uploaded/files/2016/07/18/1de4c357-c932-4c84-962f-4209a5b679bc__offers-old.json
NEW
  
https://reviews.apache.org/media/uploaded/files/2016/07/19/799bcd1f-f9c8-4b6e-bbaa-ce8022b1dac1__offers-new.json


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-18 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review142681
---



@ReviewBot retry

- Mehrdad Nurolahzade


On July 8, 2016, 4:15 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated July 8, 2016, 4:15 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1710
> https://issues.apache.org/jira/browse/AURORA-1710
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 29d224d5596eb060356f1343cf347db79b1ad687 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50052: AURORA-1736 Display reservations and persistent volumes in /offers debug http endpoint

2016-07-19 Thread Mehrdad Nurolahzade



> On July 19, 2016, 12:17 p.m., Stephan Erb wrote:
> > src/test/java/org/apache/aurora/scheduler/http/OffersTest.java, lines 
> > 143-149
> > <https://reviews.apache.org/r/50052/diff/3/?file=1447038#file1447038line143>
> >
> > The test is very closly following implementation of the code we want to 
> > test. 
> > 
> > Even though it comes with boilerplate, I believe we gain clarity if you 
> > compare the request result to a plain text json that we keep as a text 
> > fixture, rather than parsing the json back into an offer.
> > 
> > I don't have a really strong opinion here, especially given that the 
> > class was untested before. Feel free to ignore if you see this otherwise.

Yeah, I thought of that too. However, now that we are rendering JSON directly 
from Mesos protobuf structures, the generated JSON can potentially change every 
time we upgrade the Mesos library dependency, hence a test relying on text 
fixtures would be fragile and can potentially break each time (which is no big 
deal, we can have the text fixture modified according to the latest protobuf 
updates).


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50052/#review142805
-------


On July 18, 2016, 7:12 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50052/
> ---
> 
> (Updated July 18, 2016, 7:12 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Stephan Erb.
> 
> 
> Bugs: AURORA-1736
> https://issues.apache.org/jira/browse/AURORA-1736
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1736 Display reservations and persistent volumes in /offers debug http 
> endpoint
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
>   src/main/java/org/apache/aurora/scheduler/http/Offers.java 
> 80f082410896a50d86c7886736caf79581f5051c 
>   src/test/java/org/apache/aurora/scheduler/http/OffersTest.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50052/diff/
> 
> 
> Testing
> ---
> 
> Manual, Jenkins, and end_to_end
> 
> 
> File Attachments
> 
> 
> CURRENT
>   
> https://reviews.apache.org/media/uploaded/files/2016/07/18/1de4c357-c932-4c84-962f-4209a5b679bc__offers-old.json
> NEW
>   
> https://reviews.apache.org/media/uploaded/files/2016/07/19/799bcd1f-f9c8-4b6e-bbaa-ce8022b1dac1__offers-new.json
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-19 Thread Mehrdad Nurolahzade



> On July 19, 2016, 12:23 p.m., Stephan Erb wrote:
> > What's the status here? Waiting for additional review after the fix?

Yes, this board has been waiting for some love quite a while.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review142810
---


On July 8, 2016, 4:15 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated July 8, 2016, 4:15 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1710
> https://issues.apache.org/jira/browse/AURORA-1710
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 29d224d5596eb060356f1343cf347db79b1ad687 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50052: AURORA-1736 Display reservations and persistent volumes in /offers debug http endpoint

2016-07-19 Thread Mehrdad Nurolahzade



> On July 19, 2016, 2:13 p.m., Dmitriy Shirchenko wrote:
> > File Attachment: NEW - offers-new.json
> > <https://reviews.apache.org/r/50052/#fcomment103>
> >
> > Why is there a double nested labels: labels array under reservation? Is 
> > this just a carry over from how Aurora gets offers from Mesos?
> > 
> > So there is a key `labels` and then underneath is there's another key 
> > `labels` whose value is an array. Seems like one of them is not necessary.
> > 
> > ```json
> > reservation: {
> > labels: {
> > labels: [
> > {
> > key: "job",
> > value: "devcluster/www-data/prod/hello"
> > }
> > ]
> > }
> > }
> > ```

This is a question we have to put to Mesos people, it's a weired design. I can 
confirm that it is not due to our rendering of their protobuf structures.

To verify I played with their ```src/tests/reservation_endpoints_tests.cpp``` 
tests and dumped their ```ReservationInfo``` structures and noticed the same 
weired nest behavior. Also, if you try to set labels through supplied http 
endpoints ```/reserve``` you would notice the call would fail unless you supply 
nested labels:
```
curl -i \
  -d slaveId="d091a635-33c8-4409-989f-9bef16e36f34-S0" \
  -d resources='[
{
  "name": "disk",
  "type": "SCALAR",
  "scalar": { "value": 128 },
  "role": "aurora-role",
 "reservation": {
   "labels": {
 "labels" : [{"key": "job", "value": 
"devcluster/www-data/prod/hello"}]
   }
 }
}
  ]' \
  -X POST http://192.168.33.7:5050/master/reserve
```


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50052/#review142830
---


On July 18, 2016, 7:12 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50052/
> ---
> 
> (Updated July 18, 2016, 7:12 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Stephan Erb.
> 
> 
> Bugs: AURORA-1736
> https://issues.apache.org/jira/browse/AURORA-1736
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1736 Display reservations and persistent volumes in /offers debug http 
> endpoint
> 
> 
> Diffs
> -
> 
>   config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
>   src/main/java/org/apache/aurora/scheduler/http/Offers.java 
> 80f082410896a50d86c7886736caf79581f5051c 
>   src/test/java/org/apache/aurora/scheduler/http/OffersTest.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50052/diff/
> 
> 
> Testing
> ---
> 
> Manual, Jenkins, and end_to_end
> 
> 
> File Attachments
> 
> 
> CURRENT
>   
> https://reviews.apache.org/media/uploaded/files/2016/07/18/1de4c357-c932-4c84-962f-4209a5b679bc__offers-old.json
> NEW
>   
> https://reviews.apache.org/media/uploaded/files/2016/07/19/799bcd1f-f9c8-4b6e-bbaa-ce8022b1dac1__offers-new.json
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50052: AURORA-1736 Display reservations and persistent volumes in /offers debug http endpoint

2016-07-20 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50052/
---

(Updated July 20, 2016, 9:26 a.m.)


Review request for Aurora, Joshua Cohen and Stephan Erb.


Changes
---

Updated release notes


Bugs: AURORA-1736
https://issues.apache.org/jira/browse/AURORA-1736


Repository: aurora


Description
---

AURORA-1736 Display reservations and persistent volumes in /offers debug http 
endpoint


Diffs (updated)
-

  RELEASE-NOTES.md 29d224d5596eb060356f1343cf347db79b1ad687 
  config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
  src/main/java/org/apache/aurora/scheduler/http/Offers.java 
80f082410896a50d86c7886736caf79581f5051c 
  src/test/java/org/apache/aurora/scheduler/http/OffersTest.java PRE-CREATION 

Diff: https://reviews.apache.org/r/50052/diff/


Testing
---

Manual, Jenkins, and end_to_end


File Attachments


CURRENT
  
https://reviews.apache.org/media/uploaded/files/2016/07/18/1de4c357-c932-4c84-962f-4209a5b679bc__offers-old.json
NEW
  
https://reviews.apache.org/media/uploaded/files/2016/07/19/799bcd1f-f9c8-4b6e-bbaa-ce8022b1dac1__offers-new.json


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-20 Thread Mehrdad Nurolahzade



> On July 19, 2016, 3:27 p.m., Stephan Erb wrote:
> > RELEASE-NOTES.md, line 13
> > <https://reviews.apache.org/r/49048/diff/7/?file=1440195#file1440195line13>
> >
> > Maybe add here explicitly that `production` is deprecated and that 
> > people should use `tier='preferred'` instead.
> > 
> > Some people are only skimming the deprecation sections when updating, 
> > so they might miss it otherwise.

This was added to release notes under `Deprecations and removals` section for 
0.14.0 release when scheduler-side backfill work was done:
```
- Deprecated `production` field in `TaskConfig` thrift struct. Use `tier` field 
to specify task
  scheduling and resource handling behavior.
```
Would you still suggest we add a deprecation entry to 0.16.0 release note?


> On July 19, 2016, 3:27 p.m., Stephan Erb wrote:
> > src/main/python/apache/aurora/client/config.py, line 122
> > <https://reviews.apache.org/r/49048/diff/7/?file=1440198#file1440198line122>
> >
> > That comment reminds me we should have solid user facing docs. This 
> > includes:
> > 
> > * a note that `production` is deprecated in favor of tier in 
> > `reference/configuration.md`
> > * according update of `features/multitenancy.md#preemption`
> > * a user facing description of the job tier. Most people have never 
> > heard of it, so we need to explain it to them. Probably a new subsection in 
> > `features/multitenancy.md` is a good place to do this.

We can probably track documentation changes under 
[AURORA-1656](https://issues.apache.org/jira/browse/AURORA-1656), allowing this 
change-set to be limited to code changes only.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review142828
---


On July 8, 2016, 4:15 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated July 8, 2016, 4:15 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1710
> https://issues.apache.org/jira/browse/AURORA-1710
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 29d224d5596eb060356f1343cf347db79b1ad687 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-20 Thread Mehrdad Nurolahzade



> On July 19, 2016, 3:27 p.m., Stephan Erb wrote:
> > src/main/python/apache/aurora/client/cli/context.py, line 143
> > <https://reviews.apache.org/r/49048/diff/7/?file=1440197#file1440197line143>
> >
> > Side stepping the `get_config` factory and re-creating the 
> > `AnnotatedAuroraConfig` does not seem optimal in my eyes.
> > 
> > Have you considered using Aurora's binding helper concept instead? Idea 
> > would be: 
> > 
> > * change the default of tier from `None` to `{{aurora.default_tier}}` 
> > (or something similar)
> > * register a binding helper 
> > https://github.com/apache/aurora/blob/528198ecbf4adde22988f6073b043e3da049486d/src/main/python/apache/aurora/client/binding_helper.py#L33
> >  
> > * let the helper automatically resolve the  binding via a call to the 
> > scheduler whenever the user has not set a custom tier
> > 
> > Backfilling of the production flag could be moved to the scheduler side.

Thanks for pointing out binding helpers, did not know about them.

The backfilling of production/tier attributes is already happening on the 
scheduler side. We did that as part of the same ticket and it has already been 
merged to master, see [rb 48559](https://reviews.apache.org/r/48559). 

To idea here is to have the backfill logic repeated on the client side so that 
if the new client is used against the old scheduler it would still correctly 
backfill tier and production settings. Perhaps we can throw away this client 
side backfilling logic in the future and allow it to be done entirely on the 
scheduler side.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review142828
---


On July 8, 2016, 4:15 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated July 8, 2016, 4:15 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1710
> https://issues.apache.org/jira/browse/AURORA-1710
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 29d224d5596eb060356f1343cf347db79b1ad687 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-20 Thread Mehrdad Nurolahzade



> On July 19, 2016, 3:27 p.m., Stephan Erb wrote:
> > src/test/python/apache/aurora/client/cli/test_cron.py, line 111
> > <https://reviews.apache.org/r/49048/diff/7/?file=1440202#file1440202line111>
> >
> > As an example of many similar test changes:
> > 
> > The additional mock calls are obscuring the original test intend. A 
> > simple workaround would be to already set a `tier` in the job. 
> > `get_tier_config` would then no longer need to be called.

Your suggested workaround does not work. 

If you read through `_get_config_with_production_and_tier()` you would notice 
that the call on `api.get_tier_configs()` is made whether or not `tier` is 
already set. The idea is even when `tier` is set, we need to ensure that the 
selection of `tier` matches that of `production` flag. For example, 
`tier=preemptible` and `production=true` should result in `production` being 
revised to `false`.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review142828
-----------


On July 20, 2016, 10:56 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated July 20, 2016, 10:56 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Stephan Erb.
> 
> 
> Bugs: AURORA-1710
> https://issues.apache.org/jira/browse/AURORA-1710
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 29d224d5596eb060356f1343cf347db79b1ad687 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50052: AURORA-1736 Display reservations and persistent volumes in /offers debug http endpoint

2016-07-20 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50052/
---

(Updated July 20, 2016, 12:28 p.m.)


Review request for Aurora, Joshua Cohen and Stephan Erb.


Bugs: AURORA-1736
https://issues.apache.org/jira/browse/AURORA-1736


Repository: aurora


Description
---

AURORA-1736 Display reservations and persistent volumes in /offers debug http 
endpoint


Diffs
-

  RELEASE-NOTES.md 29d224d5596eb060356f1343cf347db79b1ad687 
  config/legacy_untested_classes.txt 1ea2183ab20cc5c6bca147bcea4e5c708d576b62 
  src/main/java/org/apache/aurora/scheduler/http/Offers.java 
80f082410896a50d86c7886736caf79581f5051c 
  src/test/java/org/apache/aurora/scheduler/http/OffersTest.java PRE-CREATION 

Diff: https://reviews.apache.org/r/50052/diff/


Testing
---

Manual, Jenkins, and end_to_end


File Attachments (updated)


CURRENT
  
https://reviews.apache.org/media/uploaded/files/2016/07/18/1de4c357-c932-4c84-962f-4209a5b679bc__offers-old.json
NEW
  
https://reviews.apache.org/media/uploaded/files/2016/07/19/799bcd1f-f9c8-4b6e-bbaa-ce8022b1dac1__offers-new.json
Rendering of labels by Mesos master /salves endpoint
  
https://reviews.apache.org/media/uploaded/files/2016/07/20/ea4b6db5-de80-4327-bfb4-6bd965ae21d1__Screen_Shot_2016-07-20_at_12.26.06_PM.png


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-25 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/
---

(Updated July 25, 2016, 8:30 a.m.)


Review request for Aurora, Joshua Cohen and Stephan Erb.


Changes
---

- Deprecation note in release notes
- Rebased


Bugs: AURORA-1710
https://issues.apache.org/jira/browse/AURORA-1710


Repository: aurora


Description
---

AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
Job configuration - CLI changes


Diffs (updated)
-

  RELEASE-NOTES.md ca98d7ab802114e025f6021d33bc251704c59088 
  src/main/python/apache/aurora/client/api/__init__.py 
68baf8fdb90cd26100159401c46c9963c24332b3 
  src/main/python/apache/aurora/client/cli/context.py 
9b1511801d031ff48b81c25688a55cb586b8ac66 
  src/main/python/apache/aurora/client/config.py 
2fc12559016d406c347adb416a5166cca31c961e 
  src/test/python/apache/aurora/client/cli/test_command_hooks.py 
2130f1fa71be02a004cdf8e476a270c81a7105d3 
  src/test/python/apache/aurora/client/cli/test_context.py 
204ca092adad8bf43c5032a02f61bf303fb0b2fc 
  src/test/python/apache/aurora/client/cli/test_create.py 
8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
  src/test/python/apache/aurora/client/cli/test_cron.py 
f3c522ed94a2d774865811ceb546bf9df083c14f 
  src/test/python/apache/aurora/client/cli/test_plugins.py 
a545fece5e2b3e0017a61e1be9ac478372b1f34d 
  src/test/python/apache/aurora/client/cli/test_restart.py 
967d560e5c7eb0ed85b215fb11d9751b8666acb5 
  src/test/python/apache/aurora/client/cli/util.py 
7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
  src/test/python/apache/aurora/client/test_config.py 
b1a3c1865819899ef19173be0f861783a2631d0a 

Diff: https://reviews.apache.org/r/49048/diff/


Testing
---

```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

*** OK (All tests passed) ***

mesos-master start/running, process 26868
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real19m46.324s
user0m1.496s
sys 0m0.774s
```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-25 Thread Mehrdad Nurolahzade



> On July 25, 2016, 8:55 a.m., Joshua Cohen wrote:
> > RELEASE-NOTES.md, line 20
> > <https://reviews.apache.org/r/49048/diff/7-8/?file=1440195#file1440195line20>
> >
> > Given that tier names are configurable by Aurora operators, should we 
> > word this differently?

Better?

The job configuration flag `production` is now deprecated. To achieve the same 
scheduling behavior that `production=true` used to provide, users should elect 
a `tier` for the job with attributes `preemptible=false` and `revocable=false`. 
For example, the `preferred` tier in the default tier configuation file 
(`tiers.json`) matches the above criteria.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/#review143387
---


On July 25, 2016, 8:30 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49048/
> ---
> 
> (Updated July 25, 2016, 8:30 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Stephan Erb.
> 
> 
> Bugs: AURORA-1710
> https://issues.apache.org/jira/browse/AURORA-1710
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
> Job configuration - CLI changes
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md ca98d7ab802114e025f6021d33bc251704c59088 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 68baf8fdb90cd26100159401c46c9963c24332b3 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9b1511801d031ff48b81c25688a55cb586b8ac66 
>   src/main/python/apache/aurora/client/config.py 
> 2fc12559016d406c347adb416a5166cca31c961e 
>   src/test/python/apache/aurora/client/cli/test_command_hooks.py 
> 2130f1fa71be02a004cdf8e476a270c81a7105d3 
>   src/test/python/apache/aurora/client/cli/test_context.py 
> 204ca092adad8bf43c5032a02f61bf303fb0b2fc 
>   src/test/python/apache/aurora/client/cli/test_create.py 
> 8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
>   src/test/python/apache/aurora/client/cli/test_cron.py 
> f3c522ed94a2d774865811ceb546bf9df083c14f 
>   src/test/python/apache/aurora/client/cli/test_plugins.py 
> a545fece5e2b3e0017a61e1be9ac478372b1f34d 
>   src/test/python/apache/aurora/client/cli/test_restart.py 
> 967d560e5c7eb0ed85b215fb11d9751b8666acb5 
>   src/test/python/apache/aurora/client/cli/util.py 
> 7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
> 
> Diff: https://reviews.apache.org/r/49048/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> *** OK (All tests passed) ***
> 
> mesos-master start/running, process 26868
> + RETCODE=0
> + restore_netrc
> + mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
> + true
> Connection to 127.0.0.1 closed.
> 
> real  19m46.324s
> user  0m1.496s
> sys   0m0.774s
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 49048: AURORA-1710 Make 'tier' required and remove support for 'production' flag in Job configuration - CLI changes

2016-07-25 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49048/
---

(Updated July 25, 2016, 9:25 a.m.)


Review request for Aurora, Joshua Cohen and Stephan Erb.


Changes
---

Reworded deprecation note in release notes


Bugs: AURORA-1710
https://issues.apache.org/jira/browse/AURORA-1710


Repository: aurora


Description
---

AURORA-1710 Make 'tier' required and remove support for 'production' flag in 
Job configuration - CLI changes


Diffs (updated)
-

  RELEASE-NOTES.md ca98d7ab802114e025f6021d33bc251704c59088 
  src/main/python/apache/aurora/client/api/__init__.py 
68baf8fdb90cd26100159401c46c9963c24332b3 
  src/main/python/apache/aurora/client/cli/context.py 
9b1511801d031ff48b81c25688a55cb586b8ac66 
  src/main/python/apache/aurora/client/config.py 
2fc12559016d406c347adb416a5166cca31c961e 
  src/test/python/apache/aurora/client/cli/test_command_hooks.py 
2130f1fa71be02a004cdf8e476a270c81a7105d3 
  src/test/python/apache/aurora/client/cli/test_context.py 
204ca092adad8bf43c5032a02f61bf303fb0b2fc 
  src/test/python/apache/aurora/client/cli/test_create.py 
8c27e2b340bb0a5fb5bcb44ef94d433e7f92c76c 
  src/test/python/apache/aurora/client/cli/test_cron.py 
f3c522ed94a2d774865811ceb546bf9df083c14f 
  src/test/python/apache/aurora/client/cli/test_plugins.py 
a545fece5e2b3e0017a61e1be9ac478372b1f34d 
  src/test/python/apache/aurora/client/cli/test_restart.py 
967d560e5c7eb0ed85b215fb11d9751b8666acb5 
  src/test/python/apache/aurora/client/cli/util.py 
7b4558ec7f0fb0fd2902591bc6a90dc15051dd6e 
  src/test/python/apache/aurora/client/test_config.py 
b1a3c1865819899ef19173be0f861783a2631d0a 

Diff: https://reviews.apache.org/r/49048/diff/


Testing
---

```
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh

*** OK (All tests passed) ***

mesos-master start/running, process 26868
+ RETCODE=0
+ restore_netrc
+ mv /home/vagrant/.netrc.bak /home/vagrant/.netrc
+ true
Connection to 127.0.0.1 closed.

real19m46.324s
user0m1.496s
sys 0m0.774s
```


Thanks,

Mehrdad Nurolahzade

Review Request 50432: AURORA-1741 Fix pystachio binding bug introduced by AURORA-1710

2016-07-25 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50432/
---

Review request for Aurora, David McLaughlin and Maxim Khutornenko.


Bugs: AURORA-1741
https://issues.apache.org/jira/browse/AURORA-1741


Repository: aurora


Description
---

AURORA-1741 Fix pystachio binding bug introduced by AURORA-1710


Diffs
-

  src/main/python/apache/aurora/client/config.py 
96cd9ddf77cac449d68537cd652bca03ef87d75d 

Diff: https://reviews.apache.org/r/50432/diff/


Testing
---

```
./build-support/jenkins/build.sh

```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 50432: AURORA-1741 Fix pystachio binding bug introduced by AURORA-1710

2016-07-26 Thread Mehrdad Nurolahzade



> On July 26, 2016, 7:36 a.m., Joshua Cohen wrote:
> > I know this is already landed, but it would be nice to follow this up with 
> > a test case if possible?

The logic has two test cases: 
(https://github.com/apache/aurora/blob/380307ac1878cb71bc86d2ccd7887192161254cf/src/test/python/apache/aurora/client/test_config.py#L228-L243).


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50432/#review143525
---


On July 25, 2016, 7:36 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50432/
> ---
> 
> (Updated July 25, 2016, 7:36 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1741
> https://issues.apache.org/jira/browse/AURORA-1741
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1741 Fix pystachio binding bug introduced by AURORA-1710
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/client/config.py 
> 96cd9ddf77cac449d68537cd652bca03ef87d75d 
> 
> Diff: https://reviews.apache.org/r/50432/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./build-support/jenkins/build.sh
> 
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50432: AURORA-1741 Fix pystachio binding bug introduced by AURORA-1710

2016-07-26 Thread Mehrdad Nurolahzade



> On July 26, 2016, 7:36 a.m., Joshua Cohen wrote:
> > I know this is already landed, but it would be nice to follow this up with 
> > a test case if possible?
> 
> Mehrdad Nurolahzade wrote:
> The logic has two test cases: 
> (https://github.com/apache/aurora/blob/380307ac1878cb71bc86d2ccd7887192161254cf/src/test/python/apache/aurora/client/test_config.py#L228-L243).
> 
> Joshua Cohen wrote:
> Yes, but they weren't failing though? It seems they were not sufficient 
> to catch this problem.

Alright, I'll add a test case with moustache variables (to prove that it works 
now with bindings).


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50432/#review143525
-------


On July 25, 2016, 7:36 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50432/
> ---
> 
> (Updated July 25, 2016, 7:36 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1741
> https://issues.apache.org/jira/browse/AURORA-1741
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1741 Fix pystachio binding bug introduced by AURORA-1710
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/client/config.py 
> 96cd9ddf77cac449d68537cd652bca03ef87d75d 
> 
> Diff: https://reviews.apache.org/r/50432/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./build-support/jenkins/build.sh
> 
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Review Request 50530: AURORA-1656 Document tier concept

2016-07-27 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50530/
---

Review request for Aurora, Maxim Khutornenko and Stephan Erb.


Bugs: AURORA-1656
https://issues.apache.org/jira/browse/AURORA-1656


Repository: aurora


Description
---

AURORA-1656 Document tier concept


Diffs
-

  docs/features/multitenancy.md 62bcd535d3bf39c9ea7e6d5958f6ae8ba0867c0d 
  docs/reference/configuration.md 64c076d862453545652fbaa2d3e29f284ddd164d 

Diff: https://reviews.apache.org/r/50530/diff/


Testing
---


Thanks,

Mehrdad Nurolahzade

Re: Review Request 50530: AURORA-1656 Document tier concept

2016-07-27 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50530/
---

(Updated July 27, 2016, 3:41 p.m.)


Review request for Aurora, Maxim Khutornenko and Stephan Erb.


Changes
---

Marked `production` as deprecated in `Job` object reference table


Bugs: AURORA-1656
https://issues.apache.org/jira/browse/AURORA-1656


Repository: aurora


Description
---

AURORA-1656 Document tier concept


Diffs (updated)
-

  docs/features/multitenancy.md 62bcd535d3bf39c9ea7e6d5958f6ae8ba0867c0d 
  docs/reference/configuration.md 64c076d862453545652fbaa2d3e29f284ddd164d 

Diff: https://reviews.apache.org/r/50530/diff/


Testing
---


Thanks,

Mehrdad Nurolahzade

Re: Review Request 50530: AURORA-1656 Document tier concept

2016-07-29 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50530/#review144167
---



Can we git this merged?

- Mehrdad Nurolahzade


On July 27, 2016, 3:41 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50530/
> ---
> 
> (Updated July 27, 2016, 3:41 p.m.)
> 
> 
> Review request for Aurora, Maxim Khutornenko and Stephan Erb.
> 
> 
> Bugs: AURORA-1656
> https://issues.apache.org/jira/browse/AURORA-1656
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1656 Document tier concept
> 
> 
> Diffs
> -
> 
>   docs/features/multitenancy.md 62bcd535d3bf39c9ea7e6d5958f6ae8ba0867c0d 
>   docs/reference/configuration.md 64c076d862453545652fbaa2d3e29f284ddd164d 
> 
> Diff: https://reviews.apache.org/r/50530/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50432: AURORA-1741 Fix pystachio binding bug introduced by AURORA-1710

2016-07-29 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50432/
---

(Updated July 29, 2016, 2 p.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Changes
---

Added unbound moustache bindings to test cases


Bugs: AURORA-1741
https://issues.apache.org/jira/browse/AURORA-1741


Repository: aurora


Description
---

AURORA-1741 Fix pystachio binding bug introduced by AURORA-1710


Diffs (updated)
-

  src/test/python/apache/aurora/client/test_config.py 
4742fa28e3156e5b20791b80f2db8392f7f2f4bf 

Diff: https://reviews.apache.org/r/50432/diff/


Testing
---

```
./build-support/jenkins/build.sh

```


Thanks,

Mehrdad Nurolahzade

Re: Review Request 50432: AURORA-1741 Fix pystachio binding bug introduced by AURORA-1710

2016-07-29 Thread Mehrdad Nurolahzade



> On July 29, 2016, 2:03 p.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/client/config.py, line 138
> > <https://reviews.apache.org/r/50432/diff/1/?file=1452163#file1452163line138>
> >
> > Isn't this undoing the bug fix to compare w/ `Empty` instead of `None`?

You are right, not sure why this is happening.

But that's not part of my new change-set, should be review board mix up!


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50432/#review144191
-------


On July 29, 2016, 2 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50432/
> ---
> 
> (Updated July 29, 2016, 2 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1741
> https://issues.apache.org/jira/browse/AURORA-1741
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1741 Fix pystachio binding bug introduced by AURORA-1710
> 
> 
> Diffs
> -
> 
>   src/test/python/apache/aurora/client/test_config.py 
> 4742fa28e3156e5b20791b80f2db8392f7f2f4bf 
> 
> Diff: https://reviews.apache.org/r/50432/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./build-support/jenkins/build.sh
> 
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50432: AURORA-1741 Fix pystachio binding bug introduced by AURORA-1710

2016-07-29 Thread Mehrdad Nurolahzade



> On July 29, 2016, 2:03 p.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/client/config.py, line 138
> > <https://reviews.apache.org/r/50432/diff/1/?file=1452163#file1452163line138>
> >
> > Isn't this undoing the bug fix to compare w/ `Empty` instead of `None`?
> 
> Mehrdad Nurolahzade wrote:
> You are right, not sure why this is happening.
> 
> But that's not part of my new change-set, should be review board mix up!

I'll create a new RB.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50432/#review144191
-----------


On July 29, 2016, 2 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50432/
> ---
> 
> (Updated July 29, 2016, 2 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1741
> https://issues.apache.org/jira/browse/AURORA-1741
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1741 Fix pystachio binding bug introduced by AURORA-1710
> 
> 
> Diffs
> -
> 
>   src/test/python/apache/aurora/client/test_config.py 
> 4742fa28e3156e5b20791b80f2db8392f7f2f4bf 
> 
> Diff: https://reviews.apache.org/r/50432/diff/
> 
> 
> Testing
> ---
> 
> ```
> ./build-support/jenkins/build.sh
> 
> ```
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Review Request 50617: AURORA-1741 Added missing test cases

2016-07-29 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50617/
---

Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Bugs: AURORA-1741
https://issues.apache.org/jira/browse/AURORA-1741


Repository: aurora


Description
---

AURORA-1741 Added missing test cases


Diffs
-

  src/test/python/apache/aurora/client/test_config.py 
4742fa28e3156e5b20791b80f2db8392f7f2f4bf 

Diff: https://reviews.apache.org/r/50617/diff/


Testing
---

`./pants test src/test/python/apache/aurora/client/cli:cli`


Thanks,

Mehrdad Nurolahzade

Re: Review Request 50617: AURORA-1741 Added missing test cases

2016-07-29 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50617/#review144212
---



Props to Joshua for finding a problem with these test scenarios. 
I'll sibmit a new version shortly.

- Mehrdad Nurolahzade


On July 29, 2016, 2:21 p.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50617/
> ---
> 
> (Updated July 29, 2016, 2:21 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1741
> https://issues.apache.org/jira/browse/AURORA-1741
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> AURORA-1741 Added missing test cases
> 
> 
> Diffs
> -
> 
>   src/test/python/apache/aurora/client/test_config.py 
> 4742fa28e3156e5b20791b80f2db8392f7f2f4bf 
> 
> Diff: https://reviews.apache.org/r/50617/diff/
> 
> 
> Testing
> ---
> 
> `./pants test src/test/python/apache/aurora/client/cli:cli`
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50617: AURORA-1741 Added missing test cases

2016-07-29 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50617/
---

(Updated July 29, 2016, 3:44 p.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Changes
---

Now using a truely unbound mustache like `{{_unbound_}}` instead of 
`{{thermos.ports[http]}}` that would not throw 
`ThermosTaskValidator.assert_all_refs_bound()` off on a `config.job()` call.


Bugs: AURORA-1741
https://issues.apache.org/jira/browse/AURORA-1741


Repository: aurora


Description
---

AURORA-1741 Added missing test cases


Diffs (updated)
-

  src/test/python/apache/aurora/client/test_config.py 
4742fa28e3156e5b20791b80f2db8392f7f2f4bf 

Diff: https://reviews.apache.org/r/50617/diff/


Testing
---

`./pants test src/test/python/apache/aurora/client/cli:cli`


Thanks,

Mehrdad Nurolahzade

Re: Review Request 50617: AURORA-1741 Added missing test cases

2016-07-29 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50617/
---

(Updated July 29, 2016, 3:46 p.m.)


Review request for Aurora, Joshua Cohen and Maxim Khutornenko.


Changes
---

Sorry, forgot to commit :)


Bugs: AURORA-1741
https://issues.apache.org/jira/browse/AURORA-1741


Repository: aurora


Description
---

AURORA-1741 Added missing test cases


Diffs (updated)
-

  src/test/python/apache/aurora/client/test_config.py 
4742fa28e3156e5b20791b80f2db8392f7f2f4bf 

Diff: https://reviews.apache.org/r/50617/diff/


Testing
---

`./pants test src/test/python/apache/aurora/client/cli:cli`


Thanks,

Mehrdad Nurolahzade

Re: Review Request 50819: Use update_job instead of creating new config object when modifying.

2016-08-04 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50819/#review144823
---


Ship it!




Ship It!

- Mehrdad Nurolahzade


On Aug. 4, 2016, 2:10 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50819/
> ---
> 
> (Updated Aug. 4, 2016, 2:10 p.m.)
> 
> 
> Review request for Aurora and Mehrdad Nurolahzade.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Use update_job instead of creating new config object when modifying. This 
> avoids losing any state (e.g. metadata) attached to the config.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/client/cli/context.py 
> 7bbfb5e815b2408b8ddaaeeb9911a5bd1b019a27 
> 
> Diff: https://reviews.apache.org/r/50819/diff/
> 
> 
> Testing
> ---
> 
> $ ./pants test.pytest src/test/python/apache/aurora/client::
> 
> 14:07:19 00:00 [main]
>(To run a reporting server: ./pants server)
> 14:07:19 00:00   [setup]
> 14:07:19 00:00 [parse]
>Executing tasks in goals: test
> 14:07:19 00:00   [test]
> 14:07:19 00:00 [test-prep-command]
> 14:07:19 00:00 [test]
> 14:07:19 00:00 [pytest]
> 14:07:19 00:00   [run]
>  
> 14:07:19 00:00 [chroot]== test session starts 
> ===
>  platform darwin -- Python 2.7.10 -- py-1.4.31 -- 
> pytest-2.6.4
>  plugins: timeout, cov
>  collected 339 items 
>  
>  
> src/test/python/apache/aurora/client/cli/test_config_noun.py ...
>  src/test/python/apache/aurora/client/cli/test_context.py 
> 
>  src/test/python/apache/aurora/client/cli/test_version.py 
> .
>  src/test/python/apache/aurora/client/cli/test_quota.py 
> .
>  src/test/python/apache/aurora/client/cli/test_plugins.py 
> .
>  src/test/python/apache/aurora/client/cli/test_client.py 
> ..
>  src/test/python/apache/aurora/client/cli/test_sla.py 
> .
>  src/test/python/apache/aurora/client/cli/test_open.py 
> .
>  src/test/python/apache/aurora/client/cli/test_supdate.py 
> .
>  src/test/python/apache/aurora/client/cli/test_restart.py 
> ..
>  src/test/python/apache/aurora/client/cli/test_status.py 
> .
>  src/test/python/apache/aurora/client/cli/test_add.py 
>  src/test/python/apache/aurora/client/cli/test_diff.py ..
>  src/test/python/apache/aurora/client/cli/test_cron.py 
> ..
>  
> src/test/python/apache/aurora/client/cli/test_command_hooks.py ..
>  src/test/python/apache/aurora/client/cli/test_options.py 
> ..
>  src/test/python/apache/aurora/client/cli/test_task.py 
> .
>  src/test/python/apache/aurora/client/cli/test_create.py 
> ..
>  src/test/python/apache/aurora/client/cli/test_kill.py 
> .
>  src/test/python/apache/aurora/client/cli/test_inspect.py 
> ...
>  
> src/test/python/apache/aurora/client/cli/test_api_from_cli.py ..
>  
> src/test/python/apache/aurora/client/cli/test_diff_formatter.py .
>  
> src/test/python/apache/aurora/client/api/test_updater_util.py ...
>  
> src/test/python/apache/aurora/client/api/test_job_monitor.py ..
>  src/test/python/apache/aurora/client/api/test_api.py 
> ..
>  
> src/test/python/apache/aurora/client/api/test_task_util.py .
>  
> src/test/python/apache/aurora/client/api/test_restarter.py ..
>  
> src/test/python/apache/aurora/client/api/test_instance_watcher.py ..
>  
> src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> .
>  src/test/python/apache/aurora/client/api/test_sla.py 
> ..
>

Review Request 50902: AURORA-1656 Fix broken links in tier documentation

2016-08-08 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50902/
---

Review request for Aurora and Maxim Khutornenko.


Bugs: AURORA-1656
https://issues.apache.org/jira/browse/AURORA-1656


Repository: aurora


Description
---

Just realized that tier documentation (that I recently touched for AURORA-1656) 
has broken links that I had not notice.


Diffs
-

  docs/features/multitenancy.md 417c433665bf9e32416bde426078883931783c1c 
  docs/reference/configuration.md 16b31be5bf3021eb2f646e7a4361d5b614bd1164 

Diff: https://reviews.apache.org/r/50902/diff/


Testing
---

Manual testing


Thanks,

Mehrdad Nurolahzade

Re: Review Request 50902: AURORA-1656 Fix broken links in tier documentation

2016-08-10 Thread Mehrdad Nurolahzade



> On Aug. 10, 2016, 10:36 a.m., Maxim Khutornenko wrote:
> > The easiest way to validate doc changes like this is to push your branch to 
> > your github remote and post a link here for review. Care to give it a try?

Will do.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50902/#review145364
---


On Aug. 8, 2016, 10:13 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50902/
> ---
> 
> (Updated Aug. 8, 2016, 10:13 a.m.)
> 
> 
> Review request for Aurora and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1656
> https://issues.apache.org/jira/browse/AURORA-1656
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Just realized that tier documentation (that I recently touched for 
> AURORA-1656) has broken links that I had not notice.
> 
> 
> Diffs
> -
> 
>   docs/features/multitenancy.md 417c433665bf9e32416bde426078883931783c1c 
>   docs/reference/configuration.md 16b31be5bf3021eb2f646e7a4361d5b614bd1164 
> 
> Diff: https://reviews.apache.org/r/50902/diff/
> 
> 
> Testing
> ---
> 
> Manual testing
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50902: AURORA-1656 Fix broken links in tier documentation

2016-08-10 Thread Mehrdad Nurolahzade



> On Aug. 10, 2016, 10:36 a.m., Maxim Khutornenko wrote:
> > The easiest way to validate doc changes like this is to push your branch to 
> > your github remote and post a link here for review. Care to give it a try?
> 
> Mehrdad Nurolahzade wrote:
> Will do.

Here is the github fork: https://github.com/nurolahzade/aurora


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50902/#review145364
---


On Aug. 8, 2016, 10:13 a.m., Mehrdad Nurolahzade wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50902/
> ---
> 
> (Updated Aug. 8, 2016, 10:13 a.m.)
> 
> 
> Review request for Aurora and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1656
> https://issues.apache.org/jira/browse/AURORA-1656
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Just realized that tier documentation (that I recently touched for 
> AURORA-1656) has broken links that I had not notice.
> 
> 
> Diffs
> -
> 
>   docs/features/multitenancy.md 417c433665bf9e32416bde426078883931783c1c 
>   docs/reference/configuration.md 16b31be5bf3021eb2f646e7a4361d5b614bd1164 
> 
> Diff: https://reviews.apache.org/r/50902/diff/
> 
> 
> Testing
> ---
> 
> Manual testing
> 
> 
> Thanks,
> 
> Mehrdad Nurolahzade
> 
>

Re: Review Request 50819: Use update_job instead of creating new config object when modifying.

2016-08-29 Thread Mehrdad Nurolahzade



> On Aug. 29, 2016, 11:01 a.m., Stephan Erb wrote:
> > @ReviewBot retry

Seems like this was already merged: 
[https://github.com/apache/aurora/commit/ec3591c812b4f7a063b3f5ee54ac66a9a4c5048b]


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50819/#review147171
---


On Aug. 4, 2016, 2:10 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50819/
> ---
> 
> (Updated Aug. 4, 2016, 2:10 p.m.)
> 
> 
> Review request for Aurora and Mehrdad Nurolahzade.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Use update_job instead of creating new config object when modifying. This 
> avoids losing any state (e.g. metadata) attached to the config.
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/client/cli/context.py 
> 7bbfb5e815b2408b8ddaaeeb9911a5bd1b019a27 
> 
> Diff: https://reviews.apache.org/r/50819/diff/
> 
> 
> Testing
> ---
> 
> $ ./pants test.pytest src/test/python/apache/aurora/client::
> 
> 14:07:19 00:00 [main]
>(To run a reporting server: ./pants server)
> 14:07:19 00:00   [setup]
> 14:07:19 00:00 [parse]
>Executing tasks in goals: test
> 14:07:19 00:00   [test]
> 14:07:19 00:00 [test-prep-command]
> 14:07:19 00:00 [test]
> 14:07:19 00:00 [pytest]
> 14:07:19 00:00   [run]
>  
> 14:07:19 00:00 [chroot]== test session starts 
> ===
>  platform darwin -- Python 2.7.10 -- py-1.4.31 -- 
> pytest-2.6.4
>  plugins: timeout, cov
>  collected 339 items 
>  
>  
> src/test/python/apache/aurora/client/cli/test_config_noun.py ...
>  src/test/python/apache/aurora/client/cli/test_context.py 
> 
>  src/test/python/apache/aurora/client/cli/test_version.py 
> .
>  src/test/python/apache/aurora/client/cli/test_quota.py 
> .
>  src/test/python/apache/aurora/client/cli/test_plugins.py 
> .
>  src/test/python/apache/aurora/client/cli/test_client.py 
> ..
>  src/test/python/apache/aurora/client/cli/test_sla.py 
> .
>  src/test/python/apache/aurora/client/cli/test_open.py 
> .
>  src/test/python/apache/aurora/client/cli/test_supdate.py 
> .
>  src/test/python/apache/aurora/client/cli/test_restart.py 
> ..
>  src/test/python/apache/aurora/client/cli/test_status.py 
> .
>  src/test/python/apache/aurora/client/cli/test_add.py 
>  src/test/python/apache/aurora/client/cli/test_diff.py ..
>  src/test/python/apache/aurora/client/cli/test_cron.py 
> ..
>  
> src/test/python/apache/aurora/client/cli/test_command_hooks.py ..
>  src/test/python/apache/aurora/client/cli/test_options.py 
> ..
>  src/test/python/apache/aurora/client/cli/test_task.py 
> .
>  src/test/python/apache/aurora/client/cli/test_create.py 
> ..
>  src/test/python/apache/aurora/client/cli/test_kill.py 
> .
>  src/test/python/apache/aurora/client/cli/test_inspect.py 
> ...
>  
> src/test/python/apache/aurora/client/cli/test_api_from_cli.py ..
>  
> src/test/python/apache/aurora/client/cli/test_diff_formatter.py .
>  
> src/test/python/apache/aurora/client/api/test_updater_util.py ...
>  
> src/test/python/apache/aurora/client/api/test_job_monitor.py ..
>  src/test/python/apache/aurora/client/api/test_api.py 
> ..
>  
> src/test/python/apache/aurora/client/api/test_task_util.py .
>  
> src/test/python/apache/aurora/client/api/test_restarter.py ..
>  
> src/test/python/apache/aurora/client/api/test_instance_watcher.py ..
>  
> src/test/python/apache/aurora/client/api/test_scheduler_client.py 
> .

Re: Review Request 52074: switching from launchTask to acceptOffers

2016-09-20 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52074/#review149714
---




src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 
(lines 50 - 54)
<https://reviews.apache.org/r/52074/#comment217400>

We should start disallowing or limiting import static. It causes poor 
readability. 

I was looking for a makeTask() method on this file.


- Mehrdad Nurolahzade


On Sept. 20, 2016, 12:14 p.m., Dmitriy Shirchenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52074/
> ---
> 
> (Updated Sept. 20, 2016, 12:14 p.m.)
> 
> 
> Review request for Aurora, Maxim Khutornenko, Mehrdad Nurolahzade, and Zameer 
> Manji.
> 
> 
> Bugs: AURORA-1776
> https://issues.apache.org/jira/browse/AURORA-1776
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Dynamic reservation requires us to use acceptOffers so this is in preparation 
> of that work. acceptOffers is the call to dynamically reserve resources.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java 
> 1bfd8dbf3aec314aaeb89b756ea3a2049df8777a 
>   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java 
> 14481468926b82a9bad3ee78c85f176e8ba893e2 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
> 41b9aab55aa439b13eba3545703ec97b44690444 
>   src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 
> 925c025d21bf1d44e0c1d319f6653ecfa8899481 
>   src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 
> a739bce1226d9435fa7d0b18e411064a4e78e49e 
> 
> Diff: https://reviews.apache.org/r/52074/diff/
> 
> 
> Testing
> ---
> 
> src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> 
> Thanks,
> 
> Dmitriy Shirchenko
> 
>

Re: Review Request 52074: switching from launchTask to acceptOffers

2016-09-20 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52074/#review149715
---


Ship it!




Ship It!

- Mehrdad Nurolahzade


On Sept. 20, 2016, 12:14 p.m., Dmitriy Shirchenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52074/
> ---
> 
> (Updated Sept. 20, 2016, 12:14 p.m.)
> 
> 
> Review request for Aurora, Maxim Khutornenko, Mehrdad Nurolahzade, and Zameer 
> Manji.
> 
> 
> Bugs: AURORA-1776
> https://issues.apache.org/jira/browse/AURORA-1776
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Dynamic reservation requires us to use acceptOffers so this is in preparation 
> of that work. acceptOffers is the call to dynamically reserve resources.
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java 
> 1bfd8dbf3aec314aaeb89b756ea3a2049df8777a 
>   src/main/java/org/apache/aurora/scheduler/mesos/Driver.java 
> 14481468926b82a9bad3ee78c85f176e8ba893e2 
>   src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java 
> 41b9aab55aa439b13eba3545703ec97b44690444 
>   src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java 
> 925c025d21bf1d44e0c1d319f6653ecfa8899481 
>   src/test/java/org/apache/aurora/scheduler/offers/OfferManagerImplTest.java 
> a739bce1226d9435fa7d0b18e411064a4e78e49e 
> 
> Diff: https://reviews.apache.org/r/52074/diff/
> 
> 
> Testing
> ---
> 
> src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> 
> Thanks,
> 
> Dmitriy Shirchenko
> 
>

Re: Review Request 53829: Make scheduling benchmarks more realistic.

2016-11-17 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53829/#review156206
---


Ship it!




Ship It!

- Mehrdad Nurolahzade


On Nov. 16, 2016, 4:36 p.m., Stephan Erb wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53829/
> ---
> 
> (Updated Nov. 16, 2016, 4:36 p.m.)
> 
> 
> Review request for Aurora, Mehrdad Nurolahzade and Zameer Manji.
> 
> 
> Bugs: AURORA-1802
> https://issues.apache.org/jira/browse/AURORA-1802
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch aims to strike a better balance between available offers, already
> launched tasks, and tasks to be scheduled. Specifically, it ensures that we
> only measure scheduling of tasks for which instances of the same job are
> already running (e.g. as needed by limit constraints).
> 
> The following setup is now the default: Given N hosts
> 
> * there are 0.25 * N tasks of job A scheduled
> * there are 0.25 * N tasks of job B scheduled
> * there are 0.50 * N free offers available
> * we try to schedule a task instance of job A
> 
> Hopefully this provides us with benchmark results closer to actual
> production performance.
> 
> Before:
> 
> SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark  
> thrpt   10  1646.245 ± 273.340  ops/s
> SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark  
> thrpt   10  12526.174 ± 814.677  ops/s
> 
> After:
> 
> SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark  
> thrpt   10  241.808 ± 51.952  ops/s
> SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark  
> thrpt   10  3447.655 ± 565.998  ops/s
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/BenchmarkSettings.java 
> 94f8b799c15601758aed1c5458ad7af79bda6bbb 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
> 1b56500260fc6539facf0ae3a25db02cc72b82db 
> 
> Diff: https://reviews.apache.org/r/53829/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Stephan Erb
> 
>

Re: Review Request 53862: Add benchmark for progressively filling a cluster.

2016-11-17 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53862/#review156243
---


Ship it!




Thanks for doing this.

- Mehrdad Nurolahzade


On Nov. 17, 2016, 11:55 a.m., Stephan Erb wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53862/
> ---
> 
> (Updated Nov. 17, 2016, 11:55 a.m.)
> 
> 
> Review request for Aurora, Mehrdad Nurolahzade and Zameer Manji.
> 
> 
> Bugs: AURORA-1802
> https://issues.apache.org/jira/browse/AURORA-1802
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> The existing benchmarks were only exercising the code paths for
> mismatching offers but not for successfully launched tasks. This new
> benchmark fills this gap.
> 
> In addition, this commit changes the default from DBTaskStore to
> MemTaskStore for all scheduling benchmarks. Without the switch all
> scheduling actions will be dominated by task store operations. This
> can (and has in the past) prevent the discovery of performance
> regressions.
> 
> MemTaskStore (now the default):
> SchedulingBenchmarks.FillClusterBenchmark.runBenchmark  thrpt   10 4.912 ± 
> 1.790  ops/s
> 
> DBTaskStore (former default):
> SchedulingBenchmarks.FillClusterBenchmark.runBenchmark  thrpt   10 0.418 ± 
> 0.076  ops/s
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
> 0510b892b67972fd15788f5debd979883c29ba6d 
>   src/jmh/java/org/apache/aurora/benchmark/Tasks.java 
> e548a09d94b1c6d550f9beec2b7120b64d576f20 
>   src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
> 7497fb96a578cff64b6484c587e29ec464969e84 
> 
> Diff: https://reviews.apache.org/r/53862/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew jmh -Pbenchmarks='SchedulingBenchmarks.FillClusterBenchmark'
> 
> 
> Thanks,
> 
> Stephan Erb
> 
>

Re: Review Request 53918: Fix performance regression in AttributeAggregate performance.

2016-11-21 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53918/#review156522
---


Ship it!




Ship It!


src/main/resources/org/apache/aurora/scheduler/storage/db/AttributeMapper.xml 
(line 78)
<https://reviews.apache.org/r/53918/#comment226709>

Remind me why are explicitly setting fetch size?


- Mehrdad Nurolahzade


On Nov. 19, 2016, 1:36 a.m., Stephan Erb wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53918/
> ---
> 
> (Updated Nov. 19, 2016, 1:36 a.m.)
> 
> 
> Review request for Aurora, Mehrdad Nurolahzade and Zameer Manji.
> 
> 
> Bugs: AURORA-1802
> https://issues.apache.org/jira/browse/AURORA-1802
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This commit ensures AttributeAggregate will only be computed if needed by
> limit constraints. This is the case in 0.16 but broken on master since the
> introduction of scheduling attempts with multiple tasks.
> 
> In order to better model the latter this patch also updates the the
> benchmarks to schedule multipe tasks per scheduleTask call.
> 
> Without the fix:
> SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark  
> thrpt   10  404.446 ± 31.252  ops/s
> SchedulingBenchmarks.FillClusterBenchmark.runBenchmark  thrpt   10  7.233 ± 
> 3.058  ops/s
> 
> With the fix:
> SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark  
> thrpt   10  432.245 ± 16.963  ops/s
> SchedulingBenchmarks.FillClusterBenchmark.runBenchmark  thrpt   10  87.560 ± 
> 14.600  ops/s
> 
> 
> Diffs
> -
> 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
> 631b2cfeec4f4ebd26d1d08fbd555f634ead3ac4 
>   src/main/java/org/apache/aurora/scheduler/filter/AttributeAggregate.java 
> f04149e99ad9940e15ddabb6b98238495bb9d32a 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/AttributeMapper.xml 
> 91c76ca0f469c5b94b1ab290b40ddf43e44ee0c9 
> 
> Diff: https://reviews.apache.org/r/53918/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew -Pq build
> 
> 
> Thanks,
> 
> Stephan Erb
> 
>

Re: Review Request 54107: changes to intercept and time mybatis invocations

2016-11-30 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54107/#review157427
---




src/main/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptor.java
 (line 59)
<https://reviews.apache.org/r/54107/#comment228037>

There is also a `queryCursor()` call. Is there any particular reason for 
excluding it?



src/main/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptor.java
 (lines 113 - 115)
<https://reviews.apache.org/r/54107/#comment228032>

Nit: This logic can be moved to `finally` block. 

On the very off chance of this being slow or causing an exception, it would 
still gurantee the invocation is made.



src/test/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptorTest.java
 (lines 71 - 72)
<https://reviews.apache.org/r/54107/#comment228033>

Nit: Expectations can also be cascaded, like 
`expect(clock.nowNanos()).andReturn(0L).andReturn(1000L);`


- Mehrdad Nurolahzade


On Nov. 29, 2016, 10:14 p.m., Reza Motamedi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54107/
> ---
> 
> (Updated Nov. 29, 2016, 10:14 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Mehrdad 
> Nurolahzade, and Santhosh Kumar Shanmugham.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> MyBatis allows us to intercept calls within the execution of a mapped 
> statement. This allows us to time various mapped statements and ultimately 
> gain more insight on the performance of the database module.
> 
> This patch introduces an interceptor on MyBatis on `updates` and `query` 
> mapped statements. I used the following convention to create name for the 
> newly collected stats:
> mybatis.<>
> 
> After interception the process is very similar to the one in 
> @Timed-interceptor. SlidingStats can be used to export interval averages, 
> total milliseconds and the event counts.
> 
> __example stats (from ./vars.json)__
> mybatis.create_tables_events 1
> mybatis.create_tables_events_per_sec 0.0
> mybatis.create_tables_nanos_per_event 0.0
> mybatis.create_tables_nanos_total 592633784
> mybatis.create_tables_nanos_total_per_sec 0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_events
>  3
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_events_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_per_event
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_total
>  2858362
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_total_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_events 
> 333
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_events_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_per_event
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_total
>  85745680
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_total_per_sec
>  0.0
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 
> e7287cec28e7b8ca978c506bfe821f261bc0ac26 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptor.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptorTest.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> 2e560c0d565689703b282391fe49dbf213ee25dc 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStoreTest.java
>  7e19454568540c14f91808635bf8dff82bb3 
> 
> Diff: https://reviews.apache.org/r/54107/diff/
> 
> 
> Testing
> ---
> 
> Tests are covered in InstrumentingInterceptorTest.
> 
> - testStatIsCreatedOnce
> Tests that each stat is created one time only.
> 
> - testInterceptMarksMetrics
> Tests that invocation is correctly intercepted and then proceeds.
> 
> 
> Thanks,
> 
> Reza Motamedi
> 
>

Re: Review Request 54107: changes to intercept and time mybatis invocations

2016-11-30 Thread Mehrdad Nurolahzade



> On Nov. 28, 2016, 8:18 p.m., Joshua Cohen wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptor.java,
> >  line 119
> > <https://reviews.apache.org/r/54107/diff/3/?file=1571610#file1571610line119>
> >
> > What is the performance impact of this interceptor? Do we want to 
> > intercept every call? Should we instead allow for some configurable 
> > sampling percentage?
> 
> Reza Motamedi wrote:
> Not sure if we can sample without intercepting. Once intercepted what is 
> done is only two calls to Clock::nowNanos and a subtraction. I really don't 
> think that it is an expensive operation.

One radical solution can be allowing to opt-out of interceptions like this 
(including @Timed interceptor) by exposing configuration arguments. 

Not a high priority right now, can be a future enhancement, if need be.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54107/#review157197
---


On Nov. 29, 2016, 10:14 p.m., Reza Motamedi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54107/
> ---
> 
> (Updated Nov. 29, 2016, 10:14 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Mehrdad 
> Nurolahzade, and Santhosh Kumar Shanmugham.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> MyBatis allows us to intercept calls within the execution of a mapped 
> statement. This allows us to time various mapped statements and ultimately 
> gain more insight on the performance of the database module.
> 
> This patch introduces an interceptor on MyBatis on `updates` and `query` 
> mapped statements. I used the following convention to create name for the 
> newly collected stats:
> mybatis.<>
> 
> After interception the process is very similar to the one in 
> @Timed-interceptor. SlidingStats can be used to export interval averages, 
> total milliseconds and the event counts.
> 
> __example stats (from ./vars.json)__
> mybatis.create_tables_events 1
> mybatis.create_tables_events_per_sec 0.0
> mybatis.create_tables_nanos_per_event 0.0
> mybatis.create_tables_nanos_total 592633784
> mybatis.create_tables_nanos_total_per_sec 0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_events
>  3
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_events_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_per_event
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_total
>  2858362
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_total_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_events 
> 333
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_events_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_per_event
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_total
>  85745680
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_total_per_sec
>  0.0
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 
> e7287cec28e7b8ca978c506bfe821f261bc0ac26 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptor.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptorTest.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> 2e560c0d565689703b282391fe49dbf213ee25dc 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStoreTest.java
>  7e19454568540c14f91808635bf8dff82bb3 
> 
> Diff: https://reviews.apache.org/r/54107/diff/
> 
> 
> Testing
> ---
> 
> Tests are covered in InstrumentingInterceptorTest.
> 
> - testStatIsCreatedOnce
> Tests that each stat is created one time only.
> 
> - testInterceptMarksMetrics
> Tests that invocation is correctly intercepted and then proceeds.
> 
> 
> Thanks,
> 
> Reza Motamedi
> 
>

Re: Review Request 54107: changes to intercept and time mybatis invocations

2016-11-30 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54107/#review157434
---




src/test/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptorTest.java
 (lines 152 - 156)
<https://reviews.apache.org/r/54107/#comment228038>

Can't we verify the actual values here?


- Mehrdad Nurolahzade


On Nov. 29, 2016, 10:14 p.m., Reza Motamedi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54107/
> ---
> 
> (Updated Nov. 29, 2016, 10:14 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Mehrdad 
> Nurolahzade, and Santhosh Kumar Shanmugham.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> MyBatis allows us to intercept calls within the execution of a mapped 
> statement. This allows us to time various mapped statements and ultimately 
> gain more insight on the performance of the database module.
> 
> This patch introduces an interceptor on MyBatis on `updates` and `query` 
> mapped statements. I used the following convention to create name for the 
> newly collected stats:
> mybatis.<>
> 
> After interception the process is very similar to the one in 
> @Timed-interceptor. SlidingStats can be used to export interval averages, 
> total milliseconds and the event counts.
> 
> __example stats (from ./vars.json)__
> mybatis.create_tables_events 1
> mybatis.create_tables_events_per_sec 0.0
> mybatis.create_tables_nanos_per_event 0.0
> mybatis.create_tables_nanos_total 592633784
> mybatis.create_tables_nanos_total_per_sec 0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_events
>  3
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_events_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_per_event
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_total
>  2858362
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_total_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_events 
> 333
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_events_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_per_event
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_total
>  85745680
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_total_per_sec
>  0.0
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 
> e7287cec28e7b8ca978c506bfe821f261bc0ac26 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptor.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptorTest.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> 2e560c0d565689703b282391fe49dbf213ee25dc 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStoreTest.java
>  7e19454568540c14f91808635bf8dff82bb3 
> 
> Diff: https://reviews.apache.org/r/54107/diff/
> 
> 
> Testing
> ---
> 
> Tests are covered in InstrumentingInterceptorTest.
> 
> - testStatIsCreatedOnce
> Tests that each stat is created one time only.
> 
> - testInterceptMarksMetrics
> Tests that invocation is correctly intercepted and then proceeds.
> 
> 
> Thanks,
> 
> Reza Motamedi
> 
>

Re: Review Request 54107: changes to intercept and time mybatis invocations

2016-12-01 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54107/#review157601
---


Ship it!




Ship It!


src/main/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptor.java
 (line 100)
<https://reviews.apache.org/r/54107/#comment228213>

One last nit: could we have this prefix (i.e., "mybatis.") extracted out as 
a constant?

Now that I am going through aurora stats, I have found that stats hardcoded 
like this are much harder to spot; having name formats extracted out always as 
constantsis a great help in faster spotting them.


- Mehrdad Nurolahzade


On Nov. 30, 2016, 5:53 p.m., Reza Motamedi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54107/
> ---
> 
> (Updated Nov. 30, 2016, 5:53 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Mehrdad 
> Nurolahzade, and Santhosh Kumar Shanmugham.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> MyBatis allows us to intercept calls within the execution of a mapped 
> statement. This allows us to time various mapped statements and ultimately 
> gain more insight on the performance of the database module.
> 
> This patch introduces an interceptor on MyBatis on `updates` and `query` 
> mapped statements. I used the following convention to create name for the 
> newly collected stats:
> mybatis.<>
> 
> After interception the process is very similar to the one in 
> @Timed-interceptor. SlidingStats can be used to export interval averages, 
> total milliseconds and the event counts.
> 
> __example stats (from ./vars.json)__
> mybatis.create_tables_events 1
> mybatis.create_tables_events_per_sec 0.0
> mybatis.create_tables_nanos_per_event 0.0
> mybatis.create_tables_nanos_total 592633784
> mybatis.create_tables_nanos_total_per_sec 0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_events
>  3
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_events_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_per_event
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_total
>  2858362
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_total_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_events 
> 333
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_events_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_per_event
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_total
>  85745680
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_total_per_sec
>  0.0
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 
> e7287cec28e7b8ca978c506bfe821f261bc0ac26 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptor.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptorTest.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> 2e560c0d565689703b282391fe49dbf213ee25dc 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStoreTest.java
>  7e19454568540c14f91808635bf8dff82bb3 
> 
> Diff: https://reviews.apache.org/r/54107/diff/
> 
> 
> Testing
> ---
> 
> Tests are covered in InstrumentingInterceptorTest.
> 
> - testStatIsCreatedOnce
> Tests that each stat is created one time only.
> 
> - testInterceptMarksMetrics
> Tests that invocation is correctly intercepted and then proceeds.
> 
> 
> Thanks,
> 
> Reza Motamedi
> 
>

Re: Review Request 54107: changes to intercept and time mybatis invocations

2016-12-01 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54107/#review157635
---




src/main/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptor.java
 (lines 90 - 94)
<https://reviews.apache.org/r/54107/#comment228252>

Sorry, another nit, we can reduce GC overhead by not generating stat name 
with every interception simply by mapping from `Method` to `SlidingStats` here 
(see `TimedInterceptor`).


- Mehrdad Nurolahzade


On Nov. 30, 2016, 5:53 p.m., Reza Motamedi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54107/
> ---
> 
> (Updated Nov. 30, 2016, 5:53 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Mehrdad 
> Nurolahzade, and Santhosh Kumar Shanmugham.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> MyBatis allows us to intercept calls within the execution of a mapped 
> statement. This allows us to time various mapped statements and ultimately 
> gain more insight on the performance of the database module.
> 
> This patch introduces an interceptor on MyBatis on `updates` and `query` 
> mapped statements. I used the following convention to create name for the 
> newly collected stats:
> mybatis.<>
> 
> After interception the process is very similar to the one in 
> @Timed-interceptor. SlidingStats can be used to export interval averages, 
> total milliseconds and the event counts.
> 
> __example stats (from ./vars.json)__
> mybatis.create_tables_events 1
> mybatis.create_tables_events_per_sec 0.0
> mybatis.create_tables_nanos_per_event 0.0
> mybatis.create_tables_nanos_total 592633784
> mybatis.create_tables_nanos_total_per_sec 0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_events
>  3
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_events_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_per_event
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_total
>  2858362
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_total_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_events 
> 333
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_events_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_per_event
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_total
>  85745680
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_total_per_sec
>  0.0
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 
> e7287cec28e7b8ca978c506bfe821f261bc0ac26 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptor.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptorTest.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> 2e560c0d565689703b282391fe49dbf213ee25dc 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStoreTest.java
>  7e19454568540c14f91808635bf8dff82bb3 
> 
> Diff: https://reviews.apache.org/r/54107/diff/
> 
> 
> Testing
> ---
> 
> Tests are covered in InstrumentingInterceptorTest.
> 
> - testStatIsCreatedOnce
> Tests that each stat is created one time only.
> 
> - testInterceptMarksMetrics
> Tests that invocation is correctly intercepted and then proceeds.
> 
> 
> Thanks,
> 
> Reza Motamedi
> 
>

Re: Review Request 54107: changes to intercept and time mybatis invocations

2016-12-01 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54107/#review157657
---


Ship it!




Thanks, appreciate it!

- Mehrdad Nurolahzade


On Dec. 1, 2016, 1:37 p.m., Reza Motamedi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54107/
> ---
> 
> (Updated Dec. 1, 2016, 1:37 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, Mehrdad 
> Nurolahzade, and Santhosh Kumar Shanmugham.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> MyBatis allows us to intercept calls within the execution of a mapped 
> statement. This allows us to time various mapped statements and ultimately 
> gain more insight on the performance of the database module.
> 
> This patch introduces an interceptor on MyBatis on `updates` and `query` 
> mapped statements. I used the following convention to create name for the 
> newly collected stats:
> mybatis.<>
> 
> After interception the process is very similar to the one in 
> @Timed-interceptor. SlidingStats can be used to export interval averages, 
> total milliseconds and the event counts.
> 
> __example stats (from ./vars.json)__
> mybatis.create_tables_events 1
> mybatis.create_tables_events_per_sec 0.0
> mybatis.create_tables_nanos_per_event 0.0
> mybatis.create_tables_nanos_total 592633784
> mybatis.create_tables_nanos_total_per_sec 0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_events
>  3
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_events_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_per_event
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_total
>  2858362
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.selectAll_nanos_total_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_events 
> 333
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_events_per_sec
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_per_event
>  0.0
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_total
>  85745680
> mybatis.org.apache.aurora.scheduler.storage.db.AttributeMapper.select_nanos_total_per_sec
>  0.0
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 
> e7287cec28e7b8ca978c506bfe821f261bc0ac26 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptor.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/InstrumentingInterceptorTest.java
>  PRE-CREATION 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> 2e560c0d565689703b282391fe49dbf213ee25dc 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStoreTest.java
>  7e19454568540c14f91808635bf8dff82bb3 
> 
> Diff: https://reviews.apache.org/r/54107/diff/
> 
> 
> Testing
> ---
> 
> Tests are covered in InstrumentingInterceptorTest.
> 
> - testStatIsCreatedOnce
> Tests that each stat is created one time only.
> 
> - testInterceptMarksMetrics
> Tests that invocation is correctly intercepted and then proceeds.
> 
> 
> Thanks,
> 
> Reza Motamedi
> 
>

Re: Review Request 54269: Improve scheduling throughput via logging changes.

2016-12-01 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54269/#review157673
---




src/main/java/org/apache/aurora/scheduler/state/TaskStateMachine.java (line 474)
<https://reviews.apache.org/r/54269/#comment228304>

Did you experiment with switching state transition logs off? 
https://github.com/apache/aurora/blob/master/commons/src/main/java/org/apache/aurora/common/util/StateMachine.java#L389


- Mehrdad Nurolahzade


On Dec. 1, 2016, 3:13 p.m., Zameer Manji wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54269/
> ---
> 
> (Updated Dec. 1, 2016, 3:13 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Stephan Erb.
> 
> 
> Bugs: AURORA-1831
> https://issues.apache.org/jira/browse/AURORA-1831
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This patch makes two logging performance changes.
> 
> First, it reduces the cost of logging by replacing the costly class and line
> patterns with the cheaper logger pattern. We lose line numbers and inner class
> information for much cheaper logging.
> 
> Before
> 
> I1201 15:08:40.560 [AsyncProcessor-0, StateMachine$Builder:389] 
> SchedulerLifecycle state machine transition LEADER_AWAITING_REGISTRATION -> 
> ACTIVE
> 
> 
> After
> 
> I1201 15:06:47.181 [AsyncProcessor-0, StateMachine] SchedulerLifecycle state 
> machine transition LEADER_AWAITING_REGISTRATION -> ACTIVE
> 
> 
> Second, it reduces the verbosity of the `TaskStateMachine` logging where it 
> logs
> the work command from the transition. I don't think there is any operator 
> value
> in logging this (unlike the task state transitions) so I have lowered it to
> debug.
> 
> Performance Before:
> 
> 
> Benchmark   (numPendingTasks)  
> (numTasksToDelete)   Mode  Cnt  Score   Error  Units
> StateManagerBenchmarks.DeleteTasksBenchmark.run   N/A 
>1000  thrpt   10  2.510 ± 0.557  ops/s
> StateManagerBenchmarks.DeleteTasksBenchmark.run   N/A 
>   1  thrpt   10  0.272 ± 0.030  ops/s
> StateManagerBenchmarks.DeleteTasksBenchmark.run   N/A 
>   5  thrpt   10  0.053 ± 0.011  ops/s
> StateManagerBenchmarks.InsertPendingTasksBenchmark.run   1000 
> N/A  thrpt   10  2.446 ± 0.698  ops/s
> StateManagerBenchmarks.InsertPendingTasksBenchmark.run  1 
> N/A  thrpt   10  0.246 ± 0.018  ops/s
> StateManagerBenchmarks.InsertPendingTasksBenchmark.run  5 
> N/A  thrpt   10  0.041 ± 0.006  ops/s
> 
> 
> Performance After:
> 
> Benchmark   (numPendingTasks)  
> (numTasksToDelete)   Mode  Cnt   Score   Error  Units
> StateManagerBenchmarks.DeleteTasksBenchmark.run   N/A 
>1000  thrpt   10  14.520 ± 5.696  ops/s
> StateManagerBenchmarks.DeleteTasksBenchmark.run   N/A 
>   1  thrpt   10   1.290 ± 0.361  ops/s
> StateManagerBenchmarks.DeleteTasksBenchmark.run   N/A 
>   5  thrpt   10   0.254 ± 0.097  ops/s
> StateManagerBenchmarks.InsertPendingTasksBenchmark.run   1000 
> N/A  thrpt5   7.303 ± 5.662  ops/s
> StateManagerBenchmarks.InsertPendingTasksBenchmark.run  1 
> N/A  thrpt5   0.726 ± 0.624  ops/s
> StateManagerBenchmarks.InsertPendingTasksBenchmark.run  5 
> N/A  thrpt5   0.124 ± 0.058  ops/s
> 
> 
> There is a performance improvement in the smaller case and no noticable
> degredation in the larger cases. I also verified on a small cluster that the
> improvements exist for the larger cases. I am unable to reduce the error bars
> locally on the `InsertPendingTasksBenchmark`. I suspect the bencmark needs to 
> be
> tweaked to be more consistent.
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/state/TaskStateMachine.java 
> 23f256a7d467c79dcd5d906f76af4f0261dfd81d 
>   src/main/resources/logback.xml 84c175cec811fd95db44fd8dfcd514385606042d 
> 
> Diff: https://reviews.apache.org/r/54269/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zameer Manji
> 
>

Re: Review Request 54439: Add support for an mttu metric (median time to updated)

2016-12-07 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54439/#review158343
---



A general side note: SLA metrics calculation is currently the most expensive 
cpu-bound operation handled by the scheduler (it can take as much as 50% master 
cpu cycles). The calculators seem like a good fit for micro-benchmarking with 
JMH.


src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java (lines 188 - 
192)
<https://reviews.apache.org/r/54439/#comment229108>

This lookup seems like something that can be improved by an index? I.e., a 
mapping from `instanceId -> IScheduledTask`.


- Mehrdad Nurolahzade


On Dec. 7, 2016, 6:36 a.m., Joshua Cohen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54439/
> ---
> 
> (Updated Dec. 7, 2016, 6:36 a.m.)
> 
> 
> Review request for Aurora, Mehrdad Nurolahzade, Santhosh Kumar Shanmugham, 
> and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> The metric is calculated from the time of the `INSTANCE_UPDATING` event to 
> the subsequent `ASSIGNED` event for the task with the same instance id that 
> matches the desired task config from the update details.
> 
> My original approach to this involved converting `GroupType` and 
> `AlgorithmType` from enums (which cannot be generic) to static classes 
> (which, of course, can). This allowed me to avoid unnecessarily passing 
> update details to the `calculate` method of `SlaAlgorithm` since it's ignored 
> in all but the one, new case. However, that ended up being a lot of churn, 
> and since it turns out we need both the task details and the update details 
> to calculate this metric, I went with the below approach. If anyone feels 
> strongly, I could go back to generics and create an container class that's 
> gives access to both the tasks and update details.
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/sla/MetricCalculator.java 
> 9a56cda809fbbcb07e6dd12c7a0feb272542491d 
>   src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java 
> 5d8d5bd8f705770979f284d26d2e932aabe707e5 
>   src/main/java/org/apache/aurora/scheduler/sla/SlaGroup.java 
> 6fbd4e962b3bb6eeb0831c810a321478fd52172c 
>   src/test/java/org/apache/aurora/scheduler/sla/MetricCalculatorTest.java 
> 953b65f28a585375e36e305dea6f9f94f99abc93 
>   src/test/java/org/apache/aurora/scheduler/sla/SlaAlgorithmTest.java 
> 2e719ac6b7aea86faa22deff2cc6b5f73135761c 
>   src/test/java/org/apache/aurora/scheduler/sla/SlaModuleTest.java 
> 341e346e794c9cf9a2789b8799f38fff900ec9b3 
>   src/test/java/org/apache/aurora/scheduler/sla/SlaTestUtil.java 
> 78f440f7546de9ed6842cb51db02b3bddc9a74ff 
>   
> src/test/java/org/apache/aurora/scheduler/storage/testing/StorageTestUtil.java
>  21d26b3930ea965487b2dec48a48a98677ba022b 
>   src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java 
> 43e32eede27bbf26363a3fd1ca34ffe6f8c01a73 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  6d0e9bc6a8040393875d4f0a88e8db9d6926a88b 
> 
> Diff: https://reviews.apache.org/r/54439/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew build -Pq
> e2e tests.
> 
> 
> Thanks,
> 
> Joshua Cohen
> 
>

Re: Review Request 54439: Add support for an mttu metric (median time to updated)

2016-12-07 Thread Mehrdad Nurolahzade



> On Dec. 7, 2016, 8:42 a.m., Mehrdad Nurolahzade wrote:
> > A general side note: SLA metrics calculation is currently the most 
> > expensive cpu-bound operation handled by the scheduler (it can take as much 
> > as 50% master cpu cycles). The calculators seem like a good fit for 
> > micro-benchmarking with JMH.
> 
> Joshua Cohen wrote:
> Do you think that's a blocker to shipping this metric, or just a general 
> nice-to-have?

Not a blocker, definetly a nice to have.


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54439/#review158343
---


On Dec. 7, 2016, 6:36 a.m., Joshua Cohen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54439/
> ---
> 
> (Updated Dec. 7, 2016, 6:36 a.m.)
> 
> 
> Review request for Aurora, Mehrdad Nurolahzade, Santhosh Kumar Shanmugham, 
> and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> The metric is calculated from the time of the `INSTANCE_UPDATING` event to 
> the subsequent `ASSIGNED` event for the task with the same instance id that 
> matches the desired task config from the update details.
> 
> My original approach to this involved converting `GroupType` and 
> `AlgorithmType` from enums (which cannot be generic) to static classes 
> (which, of course, can). This allowed me to avoid unnecessarily passing 
> update details to the `calculate` method of `SlaAlgorithm` since it's ignored 
> in all but the one, new case. However, that ended up being a lot of churn, 
> and since it turns out we need both the task details and the update details 
> to calculate this metric, I went with the below approach. If anyone feels 
> strongly, I could go back to generics and create an container class that's 
> gives access to both the tasks and update details.
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/sla/MetricCalculator.java 
> 9a56cda809fbbcb07e6dd12c7a0feb272542491d 
>   src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java 
> 5d8d5bd8f705770979f284d26d2e932aabe707e5 
>   src/main/java/org/apache/aurora/scheduler/sla/SlaGroup.java 
> 6fbd4e962b3bb6eeb0831c810a321478fd52172c 
>   src/test/java/org/apache/aurora/scheduler/sla/MetricCalculatorTest.java 
> 953b65f28a585375e36e305dea6f9f94f99abc93 
>   src/test/java/org/apache/aurora/scheduler/sla/SlaAlgorithmTest.java 
> 2e719ac6b7aea86faa22deff2cc6b5f73135761c 
>   src/test/java/org/apache/aurora/scheduler/sla/SlaModuleTest.java 
> 341e346e794c9cf9a2789b8799f38fff900ec9b3 
>   src/test/java/org/apache/aurora/scheduler/sla/SlaTestUtil.java 
> 78f440f7546de9ed6842cb51db02b3bddc9a74ff 
>   
> src/test/java/org/apache/aurora/scheduler/storage/testing/StorageTestUtil.java
>  21d26b3930ea965487b2dec48a48a98677ba022b 
>   src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java 
> 43e32eede27bbf26363a3fd1ca34ffe6f8c01a73 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  6d0e9bc6a8040393875d4f0a88e8db9d6926a88b 
> 
> Diff: https://reviews.apache.org/r/54439/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew build -Pq
> e2e tests.
> 
> 
> Thanks,
> 
> Joshua Cohen
> 
>

Re: Review Request 54439: Add support for an mttu metric (median time to updated)

2016-12-07 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54439/#review158360
---




src/test/java/org/apache/aurora/scheduler/sla/SlaAlgorithmTest.java (line 261)
<https://reviews.apache.org/r/54439/#comment229125>

I find the calculation logic in some of these test cases difficult to 
follow. 

Could you maybe add comments explaining in plain math what is the 
justification behind the hard-coded expected values?


- Mehrdad Nurolahzade


On Dec. 7, 2016, 9:50 a.m., Joshua Cohen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54439/
> ---
> 
> (Updated Dec. 7, 2016, 9:50 a.m.)
> 
> 
> Review request for Aurora, Mehrdad Nurolahzade, Santhosh Kumar Shanmugham, 
> and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> The metric is calculated from the time of the `INSTANCE_UPDATING` event to 
> the subsequent `ASSIGNED` event for the task with the same instance id that 
> matches the desired task config from the update details.
> 
> My original approach to this involved converting `GroupType` and 
> `AlgorithmType` from enums (which cannot be generic) to static classes 
> (which, of course, can). This allowed me to avoid unnecessarily passing 
> update details to the `calculate` method of `SlaAlgorithm` since it's ignored 
> in all but the one, new case. However, that ended up being a lot of churn, 
> and since it turns out we need both the task details and the update details 
> to calculate this metric, I went with the below approach. If anyone feels 
> strongly, I could go back to generics and create an container class that's 
> gives access to both the tasks and update details.
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/sla/MetricCalculator.java 
> 9a56cda809fbbcb07e6dd12c7a0feb272542491d 
>   src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java 
> 5d8d5bd8f705770979f284d26d2e932aabe707e5 
>   src/main/java/org/apache/aurora/scheduler/sla/SlaGroup.java 
> 6fbd4e962b3bb6eeb0831c810a321478fd52172c 
>   src/test/java/org/apache/aurora/scheduler/sla/MetricCalculatorTest.java 
> 953b65f28a585375e36e305dea6f9f94f99abc93 
>   src/test/java/org/apache/aurora/scheduler/sla/SlaAlgorithmTest.java 
> 2e719ac6b7aea86faa22deff2cc6b5f73135761c 
>   src/test/java/org/apache/aurora/scheduler/sla/SlaModuleTest.java 
> 341e346e794c9cf9a2789b8799f38fff900ec9b3 
>   src/test/java/org/apache/aurora/scheduler/sla/SlaTestUtil.java 
> 78f440f7546de9ed6842cb51db02b3bddc9a74ff 
>   
> src/test/java/org/apache/aurora/scheduler/storage/testing/StorageTestUtil.java
>  21d26b3930ea965487b2dec48a48a98677ba022b 
>   src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java 
> 43e32eede27bbf26363a3fd1ca34ffe6f8c01a73 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  6d0e9bc6a8040393875d4f0a88e8db9d6926a88b 
> 
> Diff: https://reviews.apache.org/r/54439/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew build -Pq
> e2e tests.
> 
> 
> Thanks,
> 
> Joshua Cohen
> 
>

Re: Review Request 54439: Add support for an mttu metric (median time to updated)

2016-12-08 Thread Mehrdad Nurolahzade



> On Dec. 7, 2016, 8:42 a.m., Mehrdad Nurolahzade wrote:
> > A general side note: SLA metrics calculation is currently the most 
> > expensive cpu-bound operation handled by the scheduler (it can take as much 
> > as 50% master cpu cycles). The calculators seem like a good fit for 
> > micro-benchmarking with JMH.
> 
> Joshua Cohen wrote:
> Do you think that's a blocker to shipping this metric, or just a general 
> nice-to-have?
> 
> Mehrdad Nurolahzade wrote:
> Not a blocker, definetly a nice to have.

I created a ticket for bechmarking of SLA metrics calculations: 
https://issues.apache.org/jira/browse/AURORA-1854


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54439/#review158343
---


On Dec. 7, 2016, 9:50 a.m., Joshua Cohen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54439/
> ---
> 
> (Updated Dec. 7, 2016, 9:50 a.m.)
> 
> 
> Review request for Aurora, Mehrdad Nurolahzade, Santhosh Kumar Shanmugham, 
> and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> The metric is calculated from the time of the `INSTANCE_UPDATING` event to 
> the subsequent `ASSIGNED` event for the task with the same instance id that 
> matches the desired task config from the update details.
> 
> My original approach to this involved converting `GroupType` and 
> `AlgorithmType` from enums (which cannot be generic) to static classes 
> (which, of course, can). This allowed me to avoid unnecessarily passing 
> update details to the `calculate` method of `SlaAlgorithm` since it's ignored 
> in all but the one, new case. However, that ended up being a lot of churn, 
> and since it turns out we need both the task details and the update details 
> to calculate this metric, I went with the below approach. If anyone feels 
> strongly, I could go back to generics and create an container class that's 
> gives access to both the tasks and update details.
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/sla/MetricCalculator.java 
> 9a56cda809fbbcb07e6dd12c7a0feb272542491d 
>   src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java 
> 5d8d5bd8f705770979f284d26d2e932aabe707e5 
>   src/main/java/org/apache/aurora/scheduler/sla/SlaGroup.java 
> 6fbd4e962b3bb6eeb0831c810a321478fd52172c 
>   src/test/java/org/apache/aurora/scheduler/sla/MetricCalculatorTest.java 
> 953b65f28a585375e36e305dea6f9f94f99abc93 
>   src/test/java/org/apache/aurora/scheduler/sla/SlaAlgorithmTest.java 
> 2e719ac6b7aea86faa22deff2cc6b5f73135761c 
>   src/test/java/org/apache/aurora/scheduler/sla/SlaModuleTest.java 
> 341e346e794c9cf9a2789b8799f38fff900ec9b3 
>   src/test/java/org/apache/aurora/scheduler/sla/SlaTestUtil.java 
> 78f440f7546de9ed6842cb51db02b3bddc9a74ff 
>   
> src/test/java/org/apache/aurora/scheduler/storage/testing/StorageTestUtil.java
>  21d26b3930ea965487b2dec48a48a98677ba022b 
>   src/test/java/org/apache/aurora/scheduler/thrift/Fixtures.java 
> 43e32eede27bbf26363a3fd1ca34ffe6f8c01a73 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  6d0e9bc6a8040393875d4f0a88e8db9d6926a88b 
> 
> Diff: https://reviews.apache.org/r/54439/diff/
> 
> 
> Testing
> ---
> 
> ./gradlew build -Pq
> e2e tests.
> 
> 
> Thanks,
> 
> Joshua Cohen
> 
>

Re: Review Request 54624: Expose stats on ZooKeeper connection state

2016-12-12 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54624/#review158871
---




src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceDiscoveryModule.java
 (lines 66 - 75)
<https://reviews.apache.org/r/54624/#comment229726>

Nit: You could have used a constant String prefix (`zk_connection_state_`) 
+ `state.name()` for naming stats here.



src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceDiscoveryModule.java
 (line 67)
<https://reviews.apache.org/r/54624/#comment229687>

`zk_connection_state_connected`?



src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceDiscoveryModule.java
 (lines 135 - 151)
<https://reviews.apache.org/r/54624/#comment229728>

I would rather prefer binary state (0/1 value) to counter-like stats here. 
See `org.apache.aurora.scheduler.SchedulerLifecycle` as an example.


- Mehrdad Nurolahzade


On Dec. 11, 2016, 5:32 p.m., Jing Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54624/
> ---
> 
> (Updated Dec. 11, 2016, 5:32 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Mehrdad Nurolahzade, Stephan Erb, 
> and Zameer Manji.
> 
> 
> Bugs: AURORA-1838
> https://issues.apache.org/jira/browse/AURORA-1838
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose stats on ZooKeeper connection state
> 
> 
> Diffs
> -
> 
>   
> src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceDiscoveryModule.java
>  999a542796858dcfe9e31601c47239189043fd52 
> 
> Diff: https://reviews.apache.org/r/54624/diff/
> 
> 
> Testing
> ---
> 
> http://192.168.33.7:8081/vars
> ```
> zk_connection_state_connected 1
> zk_connection_state_lost 0
> zk_connection_state_readonly 0
> zk_connection_state_reconnected 0
> zk_connection_state_suspended 0
> zk_connection_state_unknown 0
> ```
> 
> 
> Thanks,
> 
> Jing Chen
> 
>

Re: Review Request 54624: Expose stats on ZooKeeper connection state

2016-12-13 Thread Mehrdad Nurolahzade



> On Dec. 12, 2016, 2:15 p.m., Mehrdad Nurolahzade wrote:
> > src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceDiscoveryModule.java,
> >  lines 135-151
> > <https://reviews.apache.org/r/54624/diff/2/?file=1581612#file1581612line135>
> >
> > I would rather prefer binary state (0/1 value) to counter-like stats 
> > here. See `org.apache.aurora.scheduler.SchedulerLifecycle` as an example.
> 
> Stephan Erb wrote:
> Binary state gauges have the disadvantage that they only show the 
> presence. This would imply we need to scrape `/vars` pretty often in order to 
> detect ZK connection problems. Counters don't have that limitation.

I agree. Perhaps I am baised here because we already have a solution for 
polling and diagraming stats internally.
So, in your world, how are going to use counters to contemplate current state?


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54624/#review158871
---


On Dec. 11, 2016, 5:32 p.m., Jing Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54624/
> ---
> 
> (Updated Dec. 11, 2016, 5:32 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Mehrdad Nurolahzade, Stephan Erb, 
> and Zameer Manji.
> 
> 
> Bugs: AURORA-1838
> https://issues.apache.org/jira/browse/AURORA-1838
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose stats on ZooKeeper connection state
> 
> 
> Diffs
> -
> 
>   
> src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceDiscoveryModule.java
>  999a542796858dcfe9e31601c47239189043fd52 
> 
> Diff: https://reviews.apache.org/r/54624/diff/
> 
> 
> Testing
> ---
> 
> http://192.168.33.7:8081/vars
> ```
> zk_connection_state_connected 1
> zk_connection_state_lost 0
> zk_connection_state_readonly 0
> zk_connection_state_reconnected 0
> zk_connection_state_suspended 0
> zk_connection_state_unknown 0
> ```
> 
> 
> Thanks,
> 
> Jing Chen
> 
>

Re: Review Request 54624: Expose stats on ZooKeeper connection state

2016-12-13 Thread Mehrdad Nurolahzade



> On Dec. 12, 2016, 2:15 p.m., Mehrdad Nurolahzade wrote:
> > src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceDiscoveryModule.java,
> >  lines 135-151
> > <https://reviews.apache.org/r/54624/diff/2/?file=1581612#file1581612line135>
> >
> > I would rather prefer binary state (0/1 value) to counter-like stats 
> > here. See `org.apache.aurora.scheduler.SchedulerLifecycle` as an example.
> 
> Stephan Erb wrote:
> Binary state gauges have the disadvantage that they only show the 
> presence. This would imply we need to scrape `/vars` pretty often in order to 
> detect ZK connection problems. Counters don't have that limitation.
> 
> Mehrdad Nurolahzade wrote:
> I agree. Perhaps I am baised here because we already have a solution for 
> polling and diagraming stats internally.
> So, in your world, how are going to use counters to contemplate current 
> state?
> 
> David McLaughlin wrote:
> We want both. A Gauge stat for current connection state and absolute 
> counters for the events (as they are now).
> 
> Jing Chen wrote:
> AFAIC, it is easier to know how many state chagnes happened from a 
> counter, can you tell me what we are able to get from the current state. 
> Thanks

Binary stats would give us connection state at sample times; like Stephan said 
above this implies that `/vars` is being sampled regularly, for example every 
second (that's what we do internally).


- Mehrdad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54624/#review158871
---


On Dec. 11, 2016, 5:32 p.m., Jing Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54624/
> -----------
> 
> (Updated Dec. 11, 2016, 5:32 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Mehrdad Nurolahzade, Stephan Erb, 
> and Zameer Manji.
> 
> 
> Bugs: AURORA-1838
> https://issues.apache.org/jira/browse/AURORA-1838
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Expose stats on ZooKeeper connection state
> 
> 
> Diffs
> -
> 
>   
> src/main/java/org/apache/aurora/scheduler/discovery/CuratorServiceDiscoveryModule.java
>  999a542796858dcfe9e31601c47239189043fd52 
> 
> Diff: https://reviews.apache.org/r/54624/diff/
> 
> 
> Testing
> ---
> 
> http://192.168.33.7:8081/vars
> ```
> zk_connection_state_connected 1
> zk_connection_state_lost 0
> zk_connection_state_readonly 0
> zk_connection_state_reconnected 0
> zk_connection_state_suspended 0
> zk_connection_state_unknown 0
> ```
> 
> 
> Thanks,
> 
> Jing Chen
> 
>

Re: Review Request 54847: Remove ignored snapshot stats. Add high-level timings on storage start-up lifecycle.

2016-12-19 Thread Mehrdad Nurolahzade


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54847/#review159617
---



This might have to do with `private` access modifier of `SnapshotField` 
interface: https://github.com/google/guice/wiki/AOP#limitations

- Mehrdad Nurolahzade


On Dec. 18, 2016, 3:51 p.m., David McLaughlin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54847/
> ---
> 
> (Updated Dec. 18, 2016, 3:51 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen, Stephan Erb, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> The stats I added around SnapshotFields are not applied because they are not 
> Guice-created instances. I've just removed them for now (adding timings 
> manually would be very repetitive here so there's probably a better 
> abstraction we can use).
> 
> I added metrics to cover two pretty large parts of the Scheduler start-up 
> time: storage.prepare and storage.start.
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java 
> 9c9218ca6294eaddeb47b274393603bf47af61a8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java 
> 239f2eb475bec20acfcc2990d5d933f5bec83ed4 
> 
> Diff: https://reviews.apache.org/r/54847/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> David McLaughlin
> 
>

1 2 3 >

1 - 100 of 248 matches

Mail list logo