Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-09-13 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review148803
---


Ship it!




Master (633948a) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Sept. 13, 2016, 11:13 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Sept. 13, 2016, 11:13 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md 23e8168bd9bf792c8171ee4a1e856b95828c9a6a 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> ca014f350fa868a03835a2973020b715cf0f9fd8 
>   src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 
> f4f8d0037751c9c2096747264c19f6292461b308 
>   src/jmh/java/org/apache/aurora/benchmark/UpdateStoreBenchmarks.java 
> e5228ae9baadb8cad1e5ce143df09426d6107583 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  26c45fd9cefe8a3e13746d9b00c289618a402999 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> f8b9690ab55fcc1cf81c9c4418aee5e5f3c42ed7 
>   src/main/python/apache/aurora/client/cli/update.py 
> a634c44468aa8e23488b8bf829c19f05f3f7e5a5 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  7935432b026921e854702c7c49eb4f159eec2b45 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  fa32750b667f435919c338e37652a768b24bc8e6 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 196df4754b553f05e50b66ad2f84271901bc9eba 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 2dfee1f6b69ca9d56afe8e1e20d150a9178206ee 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-09-13 Thread Santhosh Kumar Shanmugham 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/
---

(Updated Sept. 13, 2016, 4:06 p.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
Khutornenko.


Changes
---

Rebasing the commit.
- Re-ran ./build-support/jenkins/build.sh and ./pants test 
src/test/python/apache/aurora/client/cli:cli


Bugs: AURORA-1711
https://issues.apache.org/jira/browse/AURORA-1711


Repository: aurora


Description
---

Allow custom metadata to be stored with the job updates. This feature
is to help case the reconciliation on job update request on the clients.

Today when a client's requests are proxied via a middle-man,
if the middle-man times out before the scheduler responds with success,
the client is left the only option of retry. In the case of updates,
since multiple updates are not allowed in parallel, the retries fail,
with INVALID_REQUEST. Although the original update is already accepted
by the scheduler and is in-progress.

With this feature, clients can reconcile the job updates,
- by marking updates them with a unique id in the metadata field
- scheduler returns the in-progress job update in its response
- client can match the client-generated ids to reconcile its state.


Diffs (updated)
-

  RELEASE-NOTES.md 23e8168bd9bf792c8171ee4a1e856b95828c9a6a 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
ca014f350fa868a03835a2973020b715cf0f9fd8 
  src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 
f4f8d0037751c9c2096747264c19f6292461b308 
  src/jmh/java/org/apache/aurora/benchmark/UpdateStoreBenchmarks.java 
e5228ae9baadb8cad1e5ce143df09426d6107583 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
  
src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
 a3b04949f1726f110638e4f93ef45947cdb9e7f8 
  
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
 PRE-CREATION 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
26c45fd9cefe8a3e13746d9b00c289618a402999 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
594bb6219294dcc77d48dcad14e2a6f9caa0c534 
  
src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
 PRE-CREATION 
  src/main/python/apache/aurora/client/api/__init__.py 
f8b9690ab55fcc1cf81c9c4418aee5e5f3c42ed7 
  src/main/python/apache/aurora/client/cli/update.py 
a634c44468aa8e23488b8bf829c19f05f3f7e5a5 
  src/main/python/apache/aurora/client/hooks/hooked_api.py 
542f165add0f1d01a782fe6d6089bff3e736fb82 
  
src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
 8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
  src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
a40830fd668aa650c22a48cbc757b45aef27e961 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 
08530397ff75081bde6f07f9d53317b5486e0da4 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
7935432b026921e854702c7c49eb4f159eec2b45 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 fa32750b667f435919c338e37652a768b24bc8e6 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
196df4754b553f05e50b66ad2f84271901bc9eba 
  src/test/python/apache/aurora/client/cli/test_supdate.py 
2dfee1f6b69ca9d56afe8e1e20d150a9178206ee 
  src/test/python/apache/aurora/client/cli/util.py 
aac9f9c802e4ad1f06cee8cf3f56749760b33af5 

Diff: https://reviews.apache.org/r/51384/diff/


Testing
---

./build-support/jenkins/build.sh
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
./pants test src/test/python/apache/aurora/client/cli:cli


Thanks,

Santhosh Kumar Shanmugham

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-09-07 Thread Santhosh Kumar Shanmugham 


> On Sept. 7, 2016, 10:36 a.m., Maxim Khutornenko wrote:
> > Ship It!

Thank you.


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review148035
---


On Sept. 6, 2016, 8:29 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Sept. 6, 2016, 8:29 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 
> f4f8d0037751c9c2096747264c19f6292461b308 
>   src/jmh/java/org/apache/aurora/benchmark/UpdateStoreBenchmarks.java 
> e5228ae9baadb8cad1e5ce143df09426d6107583 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/update.py 
> d23243dcf93dd82f66b4c8cc4342bd2c8d2adc79 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  a7d1f74acdfe5f58eb822a4d826e920cad53dced 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-09-07 Thread Maxim Khutornenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review148035
---


Ship it!




Ship It!

- Maxim Khutornenko


On Sept. 7, 2016, 3:29 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Sept. 7, 2016, 3:29 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 
> f4f8d0037751c9c2096747264c19f6292461b308 
>   src/jmh/java/org/apache/aurora/benchmark/UpdateStoreBenchmarks.java 
> e5228ae9baadb8cad1e5ce143df09426d6107583 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/update.py 
> d23243dcf93dd82f66b4c8cc4342bd2c8d2adc79 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  a7d1f74acdfe5f58eb822a4d826e920cad53dced 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-09-07 Thread Maxim Khutornenko 


> On Sept. 2, 2016, 9:37 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java,
> >  line 913
> > 
> >
> > This will be a null ref if scheduler is called from a v-1 client that 
> > does not know anything about update metadata. You need to do something like 
> > this:
> > ```
> > .setMetadata(request.isSetMetadata() 
> > ? toBuildersSet(request.getMetadata()) 
> > : ImmutableSet.of())
> > ```
> 
> Santhosh Kumar Shanmugham wrote:
> Thrift does not expose isSetMetadata() method for optional fileds of 
> primitive type in the IJobUpdateRequest. It is available however in 
> JobUpdateRequest.java and the earlier code that converts from 
> JobUpdateRequest to IJobUpdateRequest does the check, via 
> IJobUpdateRequest.build().
> 
> // SchedulerThrift.java
>   request = IJobUpdateRequest.build(new 
> JobUpdateRequest(mutableRequest).setTaskConfig(
>   configurationManager.validateAndPopulate(
>   
> ITaskConfig.build(mutableRequest.getTaskConfig())).newBuilder()));
>   
> // IJobUpdateRequest.java
>   private IJobUpdateRequest(JobUpdateRequest wrapped) {
> ...
> this.metadata = wrapped.isSetMetadata()
> ? FluentIterable.from(wrapped.getMetadata())
>   .transform(IMetadata::build)
>   .toSet()
> : ImmutableSet.of();
>   }
>   
>   public static IJobUpdateRequest build(JobUpdateRequest wrapped) {
> return new IJobUpdateRequest(wrapped);
>   }

That's right, I was mistakenly assuming that was a `JobUpdateRequest` but that 
is tracked under `mutableRequest`.


- Maxim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147694
---


On Sept. 7, 2016, 3:29 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Sept. 7, 2016, 3:29 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 
> f4f8d0037751c9c2096747264c19f6292461b308 
>   src/jmh/java/org/apache/aurora/benchmark/UpdateStoreBenchmarks.java 
> e5228ae9baadb8cad1e5ce143df09426d6107583 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/update.py 
> d23243dcf93dd82f66b4c8cc4342bd2c8d2adc79 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-09-07 Thread Santhosh Kumar Shanmugham 


> On Sept. 2, 2016, 2:37 p.m., Maxim Khutornenko wrote:
> > api/src/main/thrift/org/apache/aurora/gen/api.thrift, line 774
> > 
> >
> > nit: all comments should be ended with '.'. Here and everywhere.

Fixed.


> On Sept. 2, 2016, 2:37 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java, 
> > line 102
> > 
> >
> > `checkNotBlank` is redundant here as you enter this block only if it's 
> > not empty.

Fixed.


> On Sept. 2, 2016, 2:37 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java,
> >  lines 932-934
> > 
> >
> > tabbing is off here

Fixed.


> On Sept. 2, 2016, 2:37 p.m., Maxim Khutornenko wrote:
> > src/main/python/apache/aurora/client/cli/context.py, line 79
> > 
> >
> > This is only used in update.py, please move it there.

Fixed.


> On Sept. 2, 2016, 2:37 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java,
> >  line 913
> > 
> >
> > This will be a null ref if scheduler is called from a v-1 client that 
> > does not know anything about update metadata. You need to do something like 
> > this:
> > ```
> > .setMetadata(request.isSetMetadata() 
> > ? toBuildersSet(request.getMetadata()) 
> > : ImmutableSet.of())
> > ```

Thrift does not expose isSetMetadata() method for optional fileds of primitive 
type in the IJobUpdateRequest. It is available however in JobUpdateRequest.java 
and the earlier code that converts from JobUpdateRequest to IJobUpdateRequest 
does the check, via IJobUpdateRequest.build().

// SchedulerThrift.java
  request = IJobUpdateRequest.build(new 
JobUpdateRequest(mutableRequest).setTaskConfig(
  configurationManager.validateAndPopulate(
  ITaskConfig.build(mutableRequest.getTaskConfig())).newBuilder()));
  
// IJobUpdateRequest.java
  private IJobUpdateRequest(JobUpdateRequest wrapped) {
...
this.metadata = wrapped.isSetMetadata()
? FluentIterable.from(wrapped.getMetadata())
  .transform(IMetadata::build)
  .toSet()
: ImmutableSet.of();
  }
  
  public static IJobUpdateRequest build(JobUpdateRequest wrapped) {
return new IJobUpdateRequest(wrapped);
  }


> On Sept. 2, 2016, 2:37 p.m., Maxim Khutornenko wrote:
> > src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml,
> >  lines 357-358
> > 
> >
> > Instead of sub-selecting from a left join, have you considered a 
> > collection with a subselect instead? See `details.updateEvents` for 
> > example. 
> > 
> > It should be more compact and most importantly, perform better on large 
> > metadata counts. You can validate the last assumption via `./gradlew jmh 
> > -Pbenchmarks='UpdateStoreBenchmarks.JobDetailsBenchmark'` or create a new 
> > benchmark to test fetching job summaries only.

Updated to use a sub-select instead of the join.

Added a benchmark test. Avoiding joins and using sub-selects appear to help out 
in the throughput. Beyond 5000 metadata pairs the throughput flattens out to 1 
op/sec.


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147694
---


On Sept. 6, 2016, 8:29 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Sept. 6, 2016, 8:29 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-09-06 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147967
---


Ship it!




Master (19866b5) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Sept. 7, 2016, 3:29 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Sept. 7, 2016, 3:29 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 
> f4f8d0037751c9c2096747264c19f6292461b308 
>   src/jmh/java/org/apache/aurora/benchmark/UpdateStoreBenchmarks.java 
> e5228ae9baadb8cad1e5ce143df09426d6107583 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/update.py 
> d23243dcf93dd82f66b4c8cc4342bd2c8d2adc79 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  a7d1f74acdfe5f58eb822a4d826e920cad53dced 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-09-06 Thread Santhosh Kumar Shanmugham 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/
---

(Updated Sept. 6, 2016, 8:29 p.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
Khutornenko.


Changes
---

Address comments from Maxim.
- Use sub-selects inplace of joins
- Add benchmark for job update metadata retrieval
- style fixes


Bugs: AURORA-1711
https://issues.apache.org/jira/browse/AURORA-1711


Repository: aurora


Description
---

Allow custom metadata to be stored with the job updates. This feature
is to help case the reconciliation on job update request on the clients.

Today when a client's requests are proxied via a middle-man,
if the middle-man times out before the scheduler responds with success,
the client is left the only option of retry. In the case of updates,
since multiple updates are not allowed in parallel, the retries fail,
with INVALID_REQUEST. Although the original update is already accepted
by the scheduler and is in-progress.

With this feature, clients can reconcile the job updates,
- by marking updates them with a unique id in the metadata field
- scheduler returns the in-progress job update in its response
- client can match the client-generated ids to reconcile its state.


Diffs (updated)
-

  RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
c5765b70501c101f0535b4eed94e9948c36808f9 
  src/jmh/java/org/apache/aurora/benchmark/JobUpdates.java 
f4f8d0037751c9c2096747264c19f6292461b308 
  src/jmh/java/org/apache/aurora/benchmark/UpdateStoreBenchmarks.java 
e5228ae9baadb8cad1e5ce143df09426d6107583 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
  
src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
 a3b04949f1726f110638e4f93ef45947cdb9e7f8 
  
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
 PRE-CREATION 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
929d021a336c6a3438613c9340c84a1096dc9069 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
594bb6219294dcc77d48dcad14e2a6f9caa0c534 
  
src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
 PRE-CREATION 
  src/main/python/apache/aurora/client/api/__init__.py 
9149c3018ae58d405f284fcbd4076d251ccc8192 
  src/main/python/apache/aurora/client/cli/update.py 
d23243dcf93dd82f66b4c8cc4342bd2c8d2adc79 
  src/main/python/apache/aurora/client/hooks/hooked_api.py 
542f165add0f1d01a782fe6d6089bff3e736fb82 
  
src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
 8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
  src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
a40830fd668aa650c22a48cbc757b45aef27e961 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 
08530397ff75081bde6f07f9d53317b5486e0da4 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
a7d1f74acdfe5f58eb822a4d826e920cad53dced 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 779dc302602ae8842084807ca868a491ea99b676 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
04551f17999d742c53dfb1b36286b119b448550e 
  src/test/python/apache/aurora/client/cli/test_supdate.py 
92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
  src/test/python/apache/aurora/client/cli/util.py 
aac9f9c802e4ad1f06cee8cf3f56749760b33af5 

Diff: https://reviews.apache.org/r/51384/diff/


Testing
---

./build-support/jenkins/build.sh
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
./pants test src/test/python/apache/aurora/client/cli:cli


Thanks,

Santhosh Kumar Shanmugham

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-09-02 Thread Maxim Khutornenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147694
---



Looks great! Only a few comments left.


api/src/main/thrift/org/apache/aurora/gen/api.thrift (line 774)


nit: all comments should be ended with '.'. Here and everywhere.



src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
(line 102)


`checkNotBlank` is redundant here as you enter this block only if it's not 
empty.



src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
(line 913)


This will be a null ref if scheduler is called from a v-1 client that does 
not know anything about update metadata. You need to do something like this:
```
.setMetadata(request.isSetMetadata() 
? toBuildersSet(request.getMetadata()) 
: ImmutableSet.of())
```



src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
(lines 932 - 934)


tabbing is off here



src/main/python/apache/aurora/client/cli/context.py (line 79)


This is only used in update.py, please move it there.



src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
 (lines 357 - 358)


Instead of sub-selecting from a left join, have you considered a collection 
with a subselect instead? See `details.updateEvents` for example. 

It should be more compact and most importantly, perform better on large 
metadata counts. You can validate the last assumption via `./gradlew jmh 
-Pbenchmarks='UpdateStoreBenchmarks.JobDetailsBenchmark'` or create a new 
benchmark to test fetching job summaries only.


- Maxim Khutornenko


On Sept. 1, 2016, 5:58 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Sept. 1, 2016, 5:58 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9cf58396dc266e97f18f761d110e564ac02bb15d 
>   src/main/python/apache/aurora/client/cli/update.py 
> d23243dcf93dd82f66b4c8cc4342bd2c8d2adc79 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-09-01 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147544
---


Ship it!




Master (bd11b1c) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Sept. 1, 2016, 5:58 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Sept. 1, 2016, 5:58 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9cf58396dc266e97f18f761d110e564ac02bb15d 
>   src/main/python/apache/aurora/client/cli/update.py 
> d23243dcf93dd82f66b4c8cc4342bd2c8d2adc79 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  a7d1f74acdfe5f58eb822a4d826e920cad53dced 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-31 Thread Santhosh Kumar Shanmugham 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/
---

(Updated Aug. 31, 2016, 10:58 p.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
Khutornenko.


Changes
---

@ReviewBot retry


Bugs: AURORA-1711
https://issues.apache.org/jira/browse/AURORA-1711


Repository: aurora


Description
---

Allow custom metadata to be stored with the job updates. This feature
is to help case the reconciliation on job update request on the clients.

Today when a client's requests are proxied via a middle-man,
if the middle-man times out before the scheduler responds with success,
the client is left the only option of retry. In the case of updates,
since multiple updates are not allowed in parallel, the retries fail,
with INVALID_REQUEST. Although the original update is already accepted
by the scheduler and is in-progress.

With this feature, clients can reconcile the job updates,
- by marking updates them with a unique id in the metadata field
- scheduler returns the in-progress job update in its response
- client can match the client-generated ids to reconcile its state.


Diffs (updated)
-

  RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
c5765b70501c101f0535b4eed94e9948c36808f9 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
  
src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
 a3b04949f1726f110638e4f93ef45947cdb9e7f8 
  
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
 PRE-CREATION 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
929d021a336c6a3438613c9340c84a1096dc9069 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
594bb6219294dcc77d48dcad14e2a6f9caa0c534 
  
src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
 PRE-CREATION 
  src/main/python/apache/aurora/client/api/__init__.py 
9149c3018ae58d405f284fcbd4076d251ccc8192 
  src/main/python/apache/aurora/client/cli/context.py 
9cf58396dc266e97f18f761d110e564ac02bb15d 
  src/main/python/apache/aurora/client/cli/update.py 
d23243dcf93dd82f66b4c8cc4342bd2c8d2adc79 
  src/main/python/apache/aurora/client/hooks/hooked_api.py 
542f165add0f1d01a782fe6d6089bff3e736fb82 
  
src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
 8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
  src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
a40830fd668aa650c22a48cbc757b45aef27e961 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 
08530397ff75081bde6f07f9d53317b5486e0da4 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
a7d1f74acdfe5f58eb822a4d826e920cad53dced 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 779dc302602ae8842084807ca868a491ea99b676 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
04551f17999d742c53dfb1b36286b119b448550e 
  src/test/python/apache/aurora/client/cli/test_supdate.py 
92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
  src/test/python/apache/aurora/client/cli/util.py 
aac9f9c802e4ad1f06cee8cf3f56749760b33af5 

Diff: https://reviews.apache.org/r/51384/diff/


Testing
---

./build-support/jenkins/build.sh
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
./pants test src/test/python/apache/aurora/client/cli:cli


Thanks,

Santhosh Kumar Shanmugham

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-31 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147543
---



Master (bd11b1c) is red with this patch.
  ./build-support/jenkins/build.sh

  Running setup.py bdist_wheel for twitter.common.log: finished with status 
'done'
  Stored in directory: 
/home/jenkins/jenkins-slave/workspace/AuroraBot/.home/.cache/pip/wheels/26/36/82/e6a6b7ca739262435cecee1cde7950bf60950f1aaa2ce21093
  Running setup.py bdist_wheel for cov-core: started
  Running setup.py bdist_wheel for cov-core: finished with status 'done'
  Stored in directory: 
/home/jenkins/jenkins-slave/workspace/AuroraBot/.home/.cache/pip/wheels/86/e1/c2/9ff8cfe9773ce07003f2c2be096e169af4614c2f634671d49b
  Running setup.py bdist_wheel for twitter.common.options: started
  Running setup.py bdist_wheel for twitter.common.options: finished with status 
'done'
  Stored in directory: 
/home/jenkins/jenkins-slave/workspace/AuroraBot/.home/.cache/pip/wheels/60/c2/40/54b323809df9598cc125f02527f93ff743cd9bd979f4a1737d
Successfully built pantsbuild.pants ansicolors setproctitle 
twitter.common.collections pathspec twitter.common.dirutil pystache scandir 
psutil pywatchman Markdown Pygments docutils twitter.common.confluence coverage 
pytest pytest-cov lmdb twitter.common.lang twitter.common.log cov-core 
twitter.common.options
Installing collected packages: ansicolors, setproctitle, twitter.common.lang, 
twitter.common.collections, six, pathspec, twitter.common.dirutil, requests, 
pystache, scandir, psutil, pywatchman, futures, setuptools, pex, Markdown, 
Pygments, docutils, twitter.common.options, twitter.common.log, 
twitter.common.confluence, monotonic, fasteners, coverage, py, pytest, 
cov-core, pytest-cov, lmdb, pantsbuild.pants
  Found existing installation: setuptools 21.2.1
Uninstalling setuptools-21.2.1:
  Successfully uninstalled setuptools-21.2.1
Successfully installed Markdown-2.1.1 Pygments-1.4 ansicolors-1.0.2 
cov-core-1.15.0 coverage-3.7.1 docutils-0.12 fasteners-0.14.1 futures-3.0.5 
lmdb-0.89 monotonic-1.2 pantsbuild.pants-1.1.0rc7 pathspec-0.3.4 pex-1.1.10 
psutil-4.3.0 py-1.4.31 pystache-0.5.3 pytest-2.6.4 pytest-cov-1.8.1 
pywatchman-1.3.0 requests-2.5.3 scandir-1.2 setproctitle-1.1.10 
setuptools-5.4.1 six-1.10.0 twitter.common.collections-0.3.7 
twitter.common.confluence-0.3.7 twitter.common.dirutil-0.3.7 
twitter.common.lang-0.3.7 twitter.common.log-0.3.7 twitter.common.options-0.3.7

05:00:27 00:00 [main]
   (To run a reporting server: ./pants server)
05:00:27 00:00   [setup]
05:00:27 00:00 [parse]
   Executing tasks in goals: compile
05:00:27 00:00   [compile]
05:00:27 00:00 [compile-prep-command]
05:00:27 00:00 [compile]
05:00:27 00:00 [python-eval]
05:00:27 00:00 [pythonstyle]
05:00:27 00:00   [cache]  
   No cached artifacts for 42 targets.
   Invalidated 42 targets.
F821:ERROR   src/main/python/apache/aurora/client/cli/update.py:194 undefined 
name 'CLIENT_UPDATE_ID'
 |  {CLIENT_UPDATE_ID: update_id})

F821:ERROR   src/main/python/apache/aurora/client/cli/update.py:230 undefined 
name 'CLIENT_UPDATE_ID'
 |   CLIENT_UPDATE_ID]


FAILURE: 2 Python Style issues found


05:00:46 00:19   [complete]
   FAILURE


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Sept. 1, 2016, 4:14 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Sept. 1, 2016, 4:14 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-31 Thread Santhosh Kumar Shanmugham 


> On Aug. 31, 2016, 6:23 p.m., Joshua Cohen wrote:
> > overall lgtm now, thanks for iterating!

Thanks


> On Aug. 31, 2016, 6:23 p.m., Joshua Cohen wrote:
> > CHANGELOG, line 4
> > 
> >
> > Changelog is automatically generated at release time. Please revert 
> > this change.

Done.


> On Aug. 31, 2016, 6:23 p.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/client/cli/options.py, line 220
> > 
> >
> > just move this to update.py since that's the only place it's used now?

Done.


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147521
---


On Aug. 31, 2016, 9:14 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 31, 2016, 9:14 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9cf58396dc266e97f18f761d110e564ac02bb15d 
>   src/main/python/apache/aurora/client/cli/update.py 
> d23243dcf93dd82f66b4c8cc4342bd2c8d2adc79 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  a7d1f74acdfe5f58eb822a4d826e920cad53dced 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-31 Thread Santhosh Kumar Shanmugham 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/
---

(Updated Aug. 31, 2016, 9:14 p.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
Khutornenko.


Changes
---

- Revert CHANGELOG
- Move CLIENT_UPDATE_ID to update.py


Bugs: AURORA-1711
https://issues.apache.org/jira/browse/AURORA-1711


Repository: aurora


Description
---

Allow custom metadata to be stored with the job updates. This feature
is to help case the reconciliation on job update request on the clients.

Today when a client's requests are proxied via a middle-man,
if the middle-man times out before the scheduler responds with success,
the client is left the only option of retry. In the case of updates,
since multiple updates are not allowed in parallel, the retries fail,
with INVALID_REQUEST. Although the original update is already accepted
by the scheduler and is in-progress.

With this feature, clients can reconcile the job updates,
- by marking updates them with a unique id in the metadata field
- scheduler returns the in-progress job update in its response
- client can match the client-generated ids to reconcile its state.


Diffs (updated)
-

  RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
c5765b70501c101f0535b4eed94e9948c36808f9 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
  
src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
 a3b04949f1726f110638e4f93ef45947cdb9e7f8 
  
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
 PRE-CREATION 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
929d021a336c6a3438613c9340c84a1096dc9069 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
594bb6219294dcc77d48dcad14e2a6f9caa0c534 
  
src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
 PRE-CREATION 
  src/main/python/apache/aurora/client/api/__init__.py 
9149c3018ae58d405f284fcbd4076d251ccc8192 
  src/main/python/apache/aurora/client/cli/context.py 
9cf58396dc266e97f18f761d110e564ac02bb15d 
  src/main/python/apache/aurora/client/cli/update.py 
d23243dcf93dd82f66b4c8cc4342bd2c8d2adc79 
  src/main/python/apache/aurora/client/hooks/hooked_api.py 
542f165add0f1d01a782fe6d6089bff3e736fb82 
  
src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
 8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
  src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
a40830fd668aa650c22a48cbc757b45aef27e961 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 
08530397ff75081bde6f07f9d53317b5486e0da4 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
a7d1f74acdfe5f58eb822a4d826e920cad53dced 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 779dc302602ae8842084807ca868a491ea99b676 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
04551f17999d742c53dfb1b36286b119b448550e 
  src/test/python/apache/aurora/client/cli/test_supdate.py 
92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
  src/test/python/apache/aurora/client/cli/util.py 
aac9f9c802e4ad1f06cee8cf3f56749760b33af5 

Diff: https://reviews.apache.org/r/51384/diff/


Testing
---

./build-support/jenkins/build.sh
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
./pants test src/test/python/apache/aurora/client/cli:cli


Thanks,

Santhosh Kumar Shanmugham

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-31 Thread Joshua Cohen 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147521
---


Ship it!




overall lgtm now, thanks for iterating!


CHANGELOG (line 4)


Changelog is automatically generated at release time. Please revert this 
change.



src/main/python/apache/aurora/client/cli/options.py (line 220)


just move this to update.py since that's the only place it's used now?


- Joshua Cohen


On Sept. 1, 2016, 12:54 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Sept. 1, 2016, 12:54 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   CHANGELOG b1dce5882bbda8d3217a8dcc1509c2a25ae55a21 
>   RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9cf58396dc266e97f18f761d110e564ac02bb15d 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> d23243dcf93dd82f66b4c8cc4342bd2c8d2adc79 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  a7d1f74acdfe5f58eb822a4d826e920cad53dced 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-31 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147522
---


Ship it!




Master (bd11b1c) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Sept. 1, 2016, 12:54 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Sept. 1, 2016, 12:54 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   CHANGELOG b1dce5882bbda8d3217a8dcc1509c2a25ae55a21 
>   RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> 9cf58396dc266e97f18f761d110e564ac02bb15d 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> d23243dcf93dd82f66b4c8cc4342bd2c8d2adc79 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  a7d1f74acdfe5f58eb822a4d826e920cad53dced 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-31 Thread Santhosh Kumar Shanmugham 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/
---

(Updated Aug. 31, 2016, 5:54 p.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
Khutornenko.


Changes
---

Address comments from @jcohen, @maxim, @zmanji, @stephanerb
- move metdata field into JobUpdateSummary
- return JobUpdateSummary inside StartJobUpdateResult (instead of the 
heavy-weight JobUpdate)
- move thrift translation of the Metadata fields into the ClientAPI layer
- update RELEASE-NOTES and CHANGELOG


Bugs: AURORA-1711
https://issues.apache.org/jira/browse/AURORA-1711


Repository: aurora


Description
---

Allow custom metadata to be stored with the job updates. This feature
is to help case the reconciliation on job update request on the clients.

Today when a client's requests are proxied via a middle-man,
if the middle-man times out before the scheduler responds with success,
the client is left the only option of retry. In the case of updates,
since multiple updates are not allowed in parallel, the retries fail,
with INVALID_REQUEST. Although the original update is already accepted
by the scheduler and is in-progress.

With this feature, clients can reconcile the job updates,
- by marking updates them with a unique id in the metadata field
- scheduler returns the in-progress job update in its response
- client can match the client-generated ids to reconcile its state.


Diffs (updated)
-

  CHANGELOG b1dce5882bbda8d3217a8dcc1509c2a25ae55a21 
  RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 
  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
c5765b70501c101f0535b4eed94e9948c36808f9 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
  
src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
 a3b04949f1726f110638e4f93ef45947cdb9e7f8 
  
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
 PRE-CREATION 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
929d021a336c6a3438613c9340c84a1096dc9069 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
594bb6219294dcc77d48dcad14e2a6f9caa0c534 
  
src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
 PRE-CREATION 
  src/main/python/apache/aurora/client/api/__init__.py 
9149c3018ae58d405f284fcbd4076d251ccc8192 
  src/main/python/apache/aurora/client/cli/context.py 
9cf58396dc266e97f18f761d110e564ac02bb15d 
  src/main/python/apache/aurora/client/cli/options.py 
1245ff15a69a4b4347672f7b556985521e813a00 
  src/main/python/apache/aurora/client/cli/update.py 
d23243dcf93dd82f66b4c8cc4342bd2c8d2adc79 
  src/main/python/apache/aurora/client/hooks/hooked_api.py 
542f165add0f1d01a782fe6d6089bff3e736fb82 
  
src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
 8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
  src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
a40830fd668aa650c22a48cbc757b45aef27e961 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 
08530397ff75081bde6f07f9d53317b5486e0da4 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
a7d1f74acdfe5f58eb822a4d826e920cad53dced 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 779dc302602ae8842084807ca868a491ea99b676 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
04551f17999d742c53dfb1b36286b119b448550e 
  src/test/python/apache/aurora/client/cli/test_supdate.py 
92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
  src/test/python/apache/aurora/client/cli/util.py 
aac9f9c802e4ad1f06cee8cf3f56749760b33af5 

Diff: https://reviews.apache.org/r/51384/diff/


Testing
---

./build-support/jenkins/build.sh
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
./pants test src/test/python/apache/aurora/client/cli:cli


Thanks,

Santhosh Kumar Shanmugham

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-31 Thread Zameer Manji 


> On Aug. 31, 2016, 2:47 p.m., Zameer Manji wrote:
> > api/src/main/thrift/org/apache/aurora/gen/api.thrift, lines 795-796
> > 
> >
> > +100
> > 
> > I don't think a client could fetch the entire JobUpdate to get the 
> > metadata it attached to the request.
> 
> Santhosh Kumar Shanmugham wrote:
> Updated the api, so that the scheduler will return a JobUpdateSummary 
> instead.
> 
> Can you elaborate on your comment - "I don't think a client could fetch 
> the entire JobUpdate to get the metadata it attached to the request."

s/could/should.

I was just echoing what Maxim said.


- Zameer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147494
---


On Aug. 27, 2016, 8:58 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 27, 2016, 8:58 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-31 Thread Santhosh Kumar Shanmugham 


> On Aug. 31, 2016, 2:47 p.m., Zameer Manji wrote:
> > api/src/main/thrift/org/apache/aurora/gen/api.thrift, lines 795-796
> > 
> >
> > +100
> > 
> > I don't think a client could fetch the entire JobUpdate to get the 
> > metadata it attached to the request.

Updated the api, so that the scheduler will return a JobUpdateSummary instead.

Can you elaborate on your comment - "I don't think a client could fetch the 
entire JobUpdate to get the metadata it attached to the request."


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147494
---


On Aug. 27, 2016, 8:58 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 27, 2016, 8:58 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-31 Thread Santhosh Kumar Shanmugham 


> On Aug. 29, 2016, 11:52 a.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/client/cli/update.py, line 193
> > 
> >
> > Am I missing something? Shouldn't this be a set of `Metadata` structs? 
> > rather than a set of strings which is what this will currently pass?
> 
> Santhosh Kumar Shanmugham wrote:
> You are totally correct. Is there any way to make the python tests do 
> type-checking?
> 
> This however makes the "assert mock_calls" on mock_api to fail. python's 
> assert, TestCase.assertEquals and testfixtures.compare all are unable to deal 
> with Metadata object. Any ideas around this would be welcome.
> 
> Joshua Cohen wrote:
> I think what I would recommend is updating `_job_update_request` to take 
> a dict of metadata key/value pairs and performing the thrift conversion 
> there. That has the benefit of encapsulating knowledge of the thrift API 
> within the API client. It also means you should be able to assert on the dict 
> that's passed in test_supdate?
> 
> That said, simple equality on the generated thrift types should work? E.g.
> 
> $ PEX_INTERPRETER=1 aurora
> >>> from gen.apache.aurora.api.ttypes import Metadata
> >>> a = Metadata('foo', 'bar')
> >>> b = Metadata('foo', 'bar')
> >>> a == b
> True
> >>> c = Metadata('baz', 'qux')
> >>> a == c
> False
> 
> Santhosh Kumar Shanmugham wrote:
> Items of a set are not getting deep equality check. Is there an 
> assertDeepEquals() ?
> 
> $ PEX_INTERPRETER=1 aurora
> >>> from gen.apache.aurora.api.ttypes import Metadata
> >>> a = Metadata('foo', 'bar')
> >>> b = Metadata('foo', 'bar')
> >>> a == b
> True
> >>> {a} == {b}
> False
> >>> set([a]) == set([b])
> False
> >>> set([a]) == set([a])
> True
> 
> Joshua Cohen wrote:
> Ahh, the thrift python compiler did not generate `__hash__` methods until 
> this commit: 
> https://github.com/apache/thrift/commit/e841b3dac619a5e5d3523d059d48db1a12e41360,
>  which does not seem to have made it into an official release yet (not that 
> it would matter if it did, since we're pinned to an older version of thrift 
> for reasons I cannot recall at the moment).
> 
> So I think you'd have to iterate the sets to check for equality (which 
> would be made more complicated by the fact that ordering is not guaranteed). 
> All the more reason to just bury the set of thrift structs internal to the 
> API impl and just pass a dict instead?

Done.


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147184
---


On Aug. 27, 2016, 8:58 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 27, 2016, 8:58 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-31 Thread Santhosh Kumar Shanmugham 


> On Aug. 29, 2016, 9:57 a.m., Maxim Khutornenko wrote:
> > api/src/main/thrift/org/apache/aurora/gen/api.thrift, lines 795-796
> > 
> >
> > How about moving this into `JobUpdateSummary` instead? As it stands 
> > now, the access to update metadata requires exposing a potentially much 
> > heavier `JobUpdate` struct through our thrift interface. 
> > 
> > Bonus point: if you move it into the `JobUpdateSummary` the metadata 
> > will be automatically exposed via the existing `getJobUpdateSummaries` RPC.
> 
> Zameer Manji wrote:
> +100
> 
> I don't think a client could fetch the entire JobUpdate to get the 
> metadata it attached to the request.

Changed the api so that the Scheduler will return a JobUpdateSummary instead 
with the client supplied metadata as part of the StartJobUpdateResult.


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147146
---


On Aug. 27, 2016, 8:58 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 27, 2016, 8:58 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-31 Thread Zameer Manji 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147494
---




api/src/main/thrift/org/apache/aurora/gen/api.thrift (lines 795 - 796)


+100

I don't think a client could fetch the entire JobUpdate to get the metadata 
it attached to the request.


- Zameer Manji


On Aug. 27, 2016, 8:58 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 27, 2016, 8:58 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-31 Thread Zameer Manji 


> On Aug. 29, 2016, 9:57 a.m., Maxim Khutornenko wrote:
> > api/src/main/thrift/org/apache/aurora/gen/api.thrift, lines 795-796
> > 
> >
> > How about moving this into `JobUpdateSummary` instead? As it stands 
> > now, the access to update metadata requires exposing a potentially much 
> > heavier `JobUpdate` struct through our thrift interface. 
> > 
> > Bonus point: if you move it into the `JobUpdateSummary` the metadata 
> > will be automatically exposed via the existing `getJobUpdateSummaries` RPC.

+100

I don't think a client could fetch the entire JobUpdate to get the metadata it 
attached to the request.


- Zameer


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147146
---


On Aug. 27, 2016, 8:58 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 27, 2016, 8:58 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-30 Thread Joshua Cohen 


> On Aug. 29, 2016, 6:52 p.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/client/cli/update.py, line 193
> > 
> >
> > Am I missing something? Shouldn't this be a set of `Metadata` structs? 
> > rather than a set of strings which is what this will currently pass?
> 
> Santhosh Kumar Shanmugham wrote:
> You are totally correct. Is there any way to make the python tests do 
> type-checking?
> 
> This however makes the "assert mock_calls" on mock_api to fail. python's 
> assert, TestCase.assertEquals and testfixtures.compare all are unable to deal 
> with Metadata object. Any ideas around this would be welcome.
> 
> Joshua Cohen wrote:
> I think what I would recommend is updating `_job_update_request` to take 
> a dict of metadata key/value pairs and performing the thrift conversion 
> there. That has the benefit of encapsulating knowledge of the thrift API 
> within the API client. It also means you should be able to assert on the dict 
> that's passed in test_supdate?
> 
> That said, simple equality on the generated thrift types should work? E.g.
> 
> $ PEX_INTERPRETER=1 aurora
> >>> from gen.apache.aurora.api.ttypes import Metadata
> >>> a = Metadata('foo', 'bar')
> >>> b = Metadata('foo', 'bar')
> >>> a == b
> True
> >>> c = Metadata('baz', 'qux')
> >>> a == c
> False
> 
> Santhosh Kumar Shanmugham wrote:
> Items of a set are not getting deep equality check. Is there an 
> assertDeepEquals() ?
> 
> $ PEX_INTERPRETER=1 aurora
> >>> from gen.apache.aurora.api.ttypes import Metadata
> >>> a = Metadata('foo', 'bar')
> >>> b = Metadata('foo', 'bar')
> >>> a == b
> True
> >>> {a} == {b}
> False
> >>> set([a]) == set([b])
> False
> >>> set([a]) == set([a])
> True

Ahh, the thrift python compiler did not generate `__hash__` methods until this 
commit: 
https://github.com/apache/thrift/commit/e841b3dac619a5e5d3523d059d48db1a12e41360,
 which does not seem to have made it into an official release yet (not that it 
would matter if it did, since we're pinned to an older version of thrift for 
reasons I cannot recall at the moment).

So I think you'd have to iterate the sets to check for equality (which would be 
made more complicated by the fact that ordering is not guaranteed). All the 
more reason to just bury the set of thrift structs internal to the API impl and 
just pass a dict instead?


- Joshua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147184
---


On Aug. 28, 2016, 3:58 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 28, 2016, 3:58 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-30 Thread Stephan Erb 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147271
---



Please don't forget to add an entry to the release notes :-)

- Stephan Erb


On Aug. 28, 2016, 5:58 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 28, 2016, 5:58 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-29 Thread Santhosh Kumar Shanmugham 


> On Aug. 29, 2016, 11:52 a.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/client/cli/update.py, line 193
> > 
> >
> > Am I missing something? Shouldn't this be a set of `Metadata` structs? 
> > rather than a set of strings which is what this will currently pass?
> 
> Santhosh Kumar Shanmugham wrote:
> You are totally correct. Is there any way to make the python tests do 
> type-checking?
> 
> This however makes the "assert mock_calls" on mock_api to fail. python's 
> assert, TestCase.assertEquals and testfixtures.compare all are unable to deal 
> with Metadata object. Any ideas around this would be welcome.
> 
> Joshua Cohen wrote:
> I think what I would recommend is updating `_job_update_request` to take 
> a dict of metadata key/value pairs and performing the thrift conversion 
> there. That has the benefit of encapsulating knowledge of the thrift API 
> within the API client. It also means you should be able to assert on the dict 
> that's passed in test_supdate?
> 
> That said, simple equality on the generated thrift types should work? E.g.
> 
> $ PEX_INTERPRETER=1 aurora
> >>> from gen.apache.aurora.api.ttypes import Metadata
> >>> a = Metadata('foo', 'bar')
> >>> b = Metadata('foo', 'bar')
> >>> a == b
> True
> >>> c = Metadata('baz', 'qux')
> >>> a == c
> False

Items of a set are not getting deep equality check. Is there an 
assertDeepEquals() ?

$ PEX_INTERPRETER=1 aurora
>>> from gen.apache.aurora.api.ttypes import Metadata
>>> a = Metadata('foo', 'bar')
>>> b = Metadata('foo', 'bar')
>>> a == b
True
>>> {a} == {b}
False
>>> set([a]) == set([b])
False
>>> set([a]) == set([a])
True


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147184
---


On Aug. 27, 2016, 8:58 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 27, 2016, 8:58 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-29 Thread Joshua Cohen 


> On Aug. 29, 2016, 6:52 p.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/client/cli/update.py, line 193
> > 
> >
> > Am I missing something? Shouldn't this be a set of `Metadata` structs? 
> > rather than a set of strings which is what this will currently pass?
> 
> Santhosh Kumar Shanmugham wrote:
> You are totally correct. Is there any way to make the python tests do 
> type-checking?
> 
> This however makes the "assert mock_calls" on mock_api to fail. python's 
> assert, TestCase.assertEquals and testfixtures.compare all are unable to deal 
> with Metadata object. Any ideas around this would be welcome.

I think what I would recommend is updating `_job_update_request` to take a dict 
of metadata key/value pairs and performing the thrift conversion there. That 
has the benefit of encapsulating knowledge of the thrift API within the API 
client. It also means you should be able to assert on the dict that's passed in 
test_supdate?

That said, simple equality on the generated thrift types should work? E.g.

$ PEX_INTERPRETER=1 aurora
>>> from gen.apache.aurora.api.ttypes import Metadata
>>> a = Metadata('foo', 'bar')
>>> b = Metadata('foo', 'bar')
>>> a == b
True
>>> c = Metadata('baz', 'qux')
>>> a == c
False


- Joshua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147184
---


On Aug. 28, 2016, 3:58 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 28, 2016, 3:58 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-29 Thread Santhosh Kumar Shanmugham 


> On Aug. 29, 2016, 11:52 a.m., Joshua Cohen wrote:
> > src/test/python/apache/aurora/client/cli/test_supdate.py, line 156
> > 
> >
> > since we're mocking this anyway, no real need to generate an actual 
> > uuid, can just have the mock return something like 'some-mocked-uuid'. 
> > That'll make it a little bit more clear, in the event of a failure, that 
> > the value was a mock and not coming from live code.

Done.


> On Aug. 29, 2016, 11:52 a.m., Joshua Cohen wrote:
> > src/main/python/apache/aurora/client/cli/update.py, line 193
> > 
> >
> > Am I missing something? Shouldn't this be a set of `Metadata` structs? 
> > rather than a set of strings which is what this will currently pass?

You are totally correct. Is there any way to make the python tests do 
type-checking?

This however makes the "assert mock_calls" on mock_api to fail. python's 
assert, TestCase.assertEquals and testfixtures.compare all are unable to deal 
with Metadata object. Any ideas around this would be welcome.


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147184
---


On Aug. 27, 2016, 8:58 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 27, 2016, 8:58 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-29 Thread Joshua Cohen 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147184
---



Looks good to me, only one issue remaining from my perspective.


src/test/python/apache/aurora/client/cli/test_supdate.py (line 153)


since we're mocking this anyway, no real need to generate an actual uuid, 
can just have the mock return something like 'some-mocked-uuid'. That'll make 
it a little bit more clear, in the event of a failure, that the value was a 
mock and not coming from live code.



src/main/python/apache/aurora/client/cli/update.py (line 193)


Am I missing something? Shouldn't this be a set of `Metadata` structs? 
rather than a set of strings which is what this will currently pass?


- Joshua Cohen


On Aug. 28, 2016, 3:58 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 28, 2016, 3:58 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-29 Thread Maxim Khutornenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147146
---




api/src/main/thrift/org/apache/aurora/gen/api.thrift (lines 795 - 796)


How about moving this into `JobUpdateSummary` instead? As it stands now, 
the access to update metadata requires exposing a potentially much heavier 
`JobUpdate` struct through our thrift interface. 

Bonus point: if you move it into the `JobUpdateSummary` the metadata will 
be automatically exposed via the existing `getJobUpdateSummaries` RPC.


- Maxim Khutornenko


On Aug. 28, 2016, 3:58 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 28, 2016, 3:58 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-27 Thread Santhosh Kumar Shanmugham 


> On Aug. 27, 2016, 6:41 p.m., Zameer Manji wrote:
> > build-support/jenkins/build.sh, line 27
> > 
> >
> > Can you please extract this out and the unrelated style fixes to a 
> > different review and add `jsirois` and I?
> > 
> > First, I don't it's a good idea to mix in these changes with your 
> > metadata change here.
> > 
> > Second, I think this is a bug that merits it's own discussion. I 
> > believe we configured pants to style checking already. It shouldn't be 
> > necessary to do this here.

Moved the checkstyle related change to a different review - 
https://reviews.apache.org/r/51484/


- Santhosh Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147091
---


On Aug. 27, 2016, 8:58 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 27, 2016, 8:58 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-27 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147092
---


Ship it!




Master (4a2aff9) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Aug. 28, 2016, 3:58 a.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 28, 2016, 3:58 a.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-27 Thread Santhosh Kumar Shanmugham 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/
---

(Updated Aug. 27, 2016, 8:58 p.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
Khutornenko.


Changes
---

Addressing comments from Zammer.
- Reverted the python check-style change in build-support/jenkins/build.sh
- Reverted the style fixes.


Bugs: AURORA-1711
https://issues.apache.org/jira/browse/AURORA-1711


Repository: aurora


Description
---

Allow custom metadata to be stored with the job updates. This feature
is to help case the reconciliation on job update request on the clients.

Today when a client's requests are proxied via a middle-man,
if the middle-man times out before the scheduler responds with success,
the client is left the only option of retry. In the case of updates,
since multiple updates are not allowed in parallel, the retries fail,
with INVALID_REQUEST. Although the original update is already accepted
by the scheduler and is in-progress.

With this feature, clients can reconcile the job updates,
- by marking updates them with a unique id in the metadata field
- scheduler returns the in-progress job update in its response
- client can match the client-generated ids to reconcile its state.


Diffs (updated)
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
c5765b70501c101f0535b4eed94e9948c36808f9 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
  
src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
 a3b04949f1726f110638e4f93ef45947cdb9e7f8 
  
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
 PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
929d021a336c6a3438613c9340c84a1096dc9069 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
594bb6219294dcc77d48dcad14e2a6f9caa0c534 
  
src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
 PRE-CREATION 
  src/main/python/apache/aurora/client/api/__init__.py 
9149c3018ae58d405f284fcbd4076d251ccc8192 
  src/main/python/apache/aurora/client/cli/context.py 
f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
  src/main/python/apache/aurora/client/cli/options.py 
1245ff15a69a4b4347672f7b556985521e813a00 
  src/main/python/apache/aurora/client/cli/update.py 
23aaa2c1b67599420408633733e4581553f7151b 
  src/main/python/apache/aurora/client/hooks/hooked_api.py 
542f165add0f1d01a782fe6d6089bff3e736fb82 
  
src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
 8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
  src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
a40830fd668aa650c22a48cbc757b45aef27e961 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 
08530397ff75081bde6f07f9d53317b5486e0da4 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 779dc302602ae8842084807ca868a491ea99b676 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
04551f17999d742c53dfb1b36286b119b448550e 
  src/test/python/apache/aurora/client/cli/test_supdate.py 
92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
  src/test/python/apache/aurora/client/cli/util.py 
aac9f9c802e4ad1f06cee8cf3f56749760b33af5 

Diff: https://reviews.apache.org/r/51384/diff/


Testing
---

./build-support/jenkins/build.sh
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
./pants test src/test/python/apache/aurora/client/cli:cli


Thanks,

Santhosh Kumar Shanmugham

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-27 Thread Zameer Manji 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147091
---




build-support/jenkins/build.sh (line 27)


Can you please extract this out and the unrelated style fixes to a 
different review and add `jsirois` and I?

First, I don't it's a good idea to mix in these changes with your metadata 
change here.

Second, I think this is a bug that merits it's own discussion. I believe we 
configured pants to style checking already. It shouldn't be necessary to do 
this here.


- Zameer Manji


On Aug. 27, 2016, 3:33 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 27, 2016, 3:33 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   build-support/jenkins/build.sh 1de144628f0e687eb3a191b0aa819bae018a8f37 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
>   src/test/python/apache/aurora/executor/common/test_sandbox.py 
> ce989b1ccda0f1bc7ba9e15dfe4be20116db3491 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-27 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review147087
---



This patch does not apply cleanly against master (4a2aff9), do you need to 
rebase?

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Aug. 27, 2016, 10:33 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 27, 2016, 10:33 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   build-support/jenkins/build.sh 1de144628f0e687eb3a191b0aa819bae018a8f37 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
>  PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
>   src/test/python/apache/aurora/executor/common/test_sandbox.py 
> ce989b1ccda0f1bc7ba9e15dfe4be20116db3491 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-27 Thread Santhosh Kumar Shanmugham 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/
---

(Updated Aug. 27, 2016, 10:33 p.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
Khutornenko.


Changes
---

Addressed @jcohen's comments.
- Dropping metadata option in the command-line
- Adding db migration
- Enable check-style and fix the style issues


Bugs: AURORA-1711
https://issues.apache.org/jira/browse/AURORA-1711


Repository: aurora


Description
---

Allow custom metadata to be stored with the job updates. This feature
is to help case the reconciliation on job update request on the clients.

Today when a client's requests are proxied via a middle-man,
if the middle-man times out before the scheduler responds with success,
the client is left the only option of retry. In the case of updates,
since multiple updates are not allowed in parallel, the retries fail,
with INVALID_REQUEST. Although the original update is already accepted
by the scheduler and is in-progress.

With this feature, clients can reconcile the job updates,
- by marking updates them with a unique id in the metadata field
- scheduler returns the in-progress job update in its response
- client can match the client-generated ids to reconcile its state.


Diffs (updated)
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
c5765b70501c101f0535b4eed94e9948c36808f9 
  build-support/jenkins/build.sh 1de144628f0e687eb3a191b0aa819bae018a8f37 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
  
src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
 a3b04949f1726f110638e4f93ef45947cdb9e7f8 
  
src/main/java/org/apache/aurora/scheduler/storage/db/migration/V008_CreateUpdateMetadataTable.java
 PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
929d021a336c6a3438613c9340c84a1096dc9069 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
594bb6219294dcc77d48dcad14e2a6f9caa0c534 
  
src/main/java/org/apache/aurora/scheduler/updater/UpdateInProgressException.java
 PRE-CREATION 
  src/main/python/apache/aurora/client/api/__init__.py 
9149c3018ae58d405f284fcbd4076d251ccc8192 
  src/main/python/apache/aurora/client/cli/context.py 
f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
  src/main/python/apache/aurora/client/cli/options.py 
1245ff15a69a4b4347672f7b556985521e813a00 
  src/main/python/apache/aurora/client/cli/update.py 
23aaa2c1b67599420408633733e4581553f7151b 
  src/main/python/apache/aurora/client/hooks/hooked_api.py 
542f165add0f1d01a782fe6d6089bff3e736fb82 
  
src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
 8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
  src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
a40830fd668aa650c22a48cbc757b45aef27e961 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 
08530397ff75081bde6f07f9d53317b5486e0da4 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 779dc302602ae8842084807ca868a491ea99b676 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
04551f17999d742c53dfb1b36286b119b448550e 
  src/test/python/apache/aurora/client/cli/test_supdate.py 
92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
  src/test/python/apache/aurora/client/cli/util.py 
aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
  src/test/python/apache/aurora/executor/common/test_sandbox.py 
ce989b1ccda0f1bc7ba9e15dfe4be20116db3491 

Diff: https://reviews.apache.org/r/51384/diff/


Testing
---

./build-support/jenkins/build.sh
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
./pants test src/test/python/apache/aurora/client/cli:cli


Thanks,

Santhosh Kumar Shanmugham

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-25 Thread Joshua Cohen 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review146835
---




src/main/java/org/apache/aurora/scheduler/updater/UpdateInprogressException.java
 (line 22)


s/inprogress/inProgress

here and below.



src/main/python/apache/aurora/client/cli/options.py (line 89)


You're going to want to pass the optional `maxsplit=1` arg here to handle 
the case where someone sets metadata to something like: foo=bar=baz.

That said, I'm not convinced we actually need to expose this as a 
user-visible option. For the time being I think it's fine for the client to 
just generate the update id and send it without user intervention.

If/when a use case for user configurable metadata arises it'll be easy 
enough to add that support then.



src/main/python/apache/aurora/client/cli/options.py (line 232)


do you think we should package-prefix this to avoid potentially conflicting 
with other metadata values in the future (i.e. 
org.apache.aurora.client.update_id)?



src/main/python/apache/aurora/client/cli/update.py (lines 218 - 226)


I feel like this method should either be renamed or be side effect free. As 
it stands, I would not expect a call to `_generated_id_if_needed` to also 
mutate the `metadata` param. So, either just return the id without setting it 
(leaving it up to the caller) or rename to something like 
`_populate_id_if_needed`?



src/main/python/apache/aurora/client/cli/update.py (line 237)


Try to avoid abbreviations in variable names, so maybe 
`start_update_result` instead?



src/main/python/apache/aurora/client/cli/update.py (line 239)


s/in/in_



src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
 (line 207)


call this prefix `jum_` for consistency with the others?



src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql (lines 312 
- 317)


You need to create a migration for this table. See 
https://github.com/apache/aurora/blob/master/docs/development/db-migration.md 
for details.



src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java (line 313)


s/Inprogress/InProgress

throughout



src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java (line 1609)


Should this be in a finally to ensure a failure here doesn't break the rest 
of the tests?



src/test/python/apache/aurora/client/cli/test_options.py (line 78)


line limit is 100 characters. I'm surprised checkstyle didn't pick this up.



src/test/python/apache/aurora/client/cli/test_supdate.py (line 156)


Looks like there's a lot of lines beyond 100 chars here as well, can you 
fix?

We'll have to figure out why checkstyle isn't calling those out as 
problematic.


- Joshua Cohen


On Aug. 24, 2016, 7:41 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 24, 2016, 7:41 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-24 Thread Aurora ReviewBot 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review146700
---


Ship it!




Master (50f47cc) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On Aug. 24, 2016, 7:41 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 24, 2016, 7:41 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInprogressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_options.py 
> f2aae5741611909d5a1700cb4da70dec313e00da 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-24 Thread Santhosh Kumar Shanmugham 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/#review146699
---



@ReviewBot retry

- Santhosh Kumar Shanmugham


On Aug. 24, 2016, 7:41 p.m., Santhosh Kumar Shanmugham wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51384/
> ---
> 
> (Updated Aug. 24, 2016, 7:41 p.m.)
> 
> 
> Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
> Khutornenko.
> 
> 
> Bugs: AURORA-1711
> https://issues.apache.org/jira/browse/AURORA-1711
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow custom metadata to be stored with the job updates. This feature
> is to help case the reconciliation on job update request on the clients.
> 
> Today when a client's requests are proxied via a middle-man,
> if the middle-man times out before the scheduler responds with success,
> the client is left the only option of retry. In the case of updates,
> since multiple updates are not allowed in parallel, the retries fail,
> with INVALID_REQUEST. Although the original update is already accepted
> by the scheduler and is in-progress.
> 
> With this feature, clients can reconcile the job updates,
> - by marking updates them with a unique id in the metadata field
> - scheduler returns the in-progress job update in its response
> - client can match the client-generated ids to reconcile its state.
> 
> 
> Diffs
> -
> 
>   api/src/main/thrift/org/apache/aurora/gen/api.thrift 
> c5765b70501c101f0535b4eed94e9948c36808f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
> d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
>   
> src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
>  a3b04949f1726f110638e4f93ef45947cdb9e7f8 
>   src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
> 78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  929d021a336c6a3438613c9340c84a1096dc9069 
>   
> src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java
>  594bb6219294dcc77d48dcad14e2a6f9caa0c534 
>   
> src/main/java/org/apache/aurora/scheduler/updater/UpdateInprogressException.java
>  PRE-CREATION 
>   src/main/python/apache/aurora/client/api/__init__.py 
> 9149c3018ae58d405f284fcbd4076d251ccc8192 
>   src/main/python/apache/aurora/client/cli/context.py 
> f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
>   src/main/python/apache/aurora/client/cli/options.py 
> 1245ff15a69a4b4347672f7b556985521e813a00 
>   src/main/python/apache/aurora/client/cli/update.py 
> 23aaa2c1b67599420408633733e4581553f7151b 
>   src/main/python/apache/aurora/client/hooks/hooked_api.py 
> 542f165add0f1d01a782fe6d6089bff3e736fb82 
>   
> src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
>  8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> a40830fd668aa650c22a48cbc757b45aef27e961 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  08530397ff75081bde6f07f9d53317b5486e0da4 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  779dc302602ae8842084807ca868a491ea99b676 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> 04551f17999d742c53dfb1b36286b119b448550e 
>   src/test/python/apache/aurora/client/cli/test_options.py 
> f2aae5741611909d5a1700cb4da70dec313e00da 
>   src/test/python/apache/aurora/client/cli/test_supdate.py 
> 92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
>   src/test/python/apache/aurora/client/cli/util.py 
> aac9f9c802e4ad1f06cee8cf3f56749760b33af5 
> 
> Diff: https://reviews.apache.org/r/51384/diff/
> 
> 
> Testing
> ---
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ./pants test src/test/python/apache/aurora/client/cli:cli
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>

Re: Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-24 Thread Santhosh Kumar Shanmugham 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/
---

(Updated Aug. 24, 2016, 7:41 p.m.)


Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
Khutornenko.


Changes
---

Fix style issues in client.


Bugs: AURORA-1711
https://issues.apache.org/jira/browse/AURORA-1711


Repository: aurora


Description
---

Allow custom metadata to be stored with the job updates. This feature
is to help case the reconciliation on job update request on the clients.

Today when a client's requests are proxied via a middle-man,
if the middle-man times out before the scheduler responds with success,
the client is left the only option of retry. In the case of updates,
since multiple updates are not allowed in parallel, the retries fail,
with INVALID_REQUEST. Although the original update is already accepted
by the scheduler and is in-progress.

With this feature, clients can reconcile the job updates,
- by marking updates them with a unique id in the metadata field
- scheduler returns the in-progress job update in its response
- client can match the client-generated ids to reconcile its state.


Diffs (updated)
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
c5765b70501c101f0535b4eed94e9948c36808f9 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
  
src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
 a3b04949f1726f110638e4f93ef45947cdb9e7f8 
  src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
929d021a336c6a3438613c9340c84a1096dc9069 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
594bb6219294dcc77d48dcad14e2a6f9caa0c534 
  
src/main/java/org/apache/aurora/scheduler/updater/UpdateInprogressException.java
 PRE-CREATION 
  src/main/python/apache/aurora/client/api/__init__.py 
9149c3018ae58d405f284fcbd4076d251ccc8192 
  src/main/python/apache/aurora/client/cli/context.py 
f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
  src/main/python/apache/aurora/client/cli/options.py 
1245ff15a69a4b4347672f7b556985521e813a00 
  src/main/python/apache/aurora/client/cli/update.py 
23aaa2c1b67599420408633733e4581553f7151b 
  src/main/python/apache/aurora/client/hooks/hooked_api.py 
542f165add0f1d01a782fe6d6089bff3e736fb82 
  
src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
 8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
  src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
a40830fd668aa650c22a48cbc757b45aef27e961 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 
08530397ff75081bde6f07f9d53317b5486e0da4 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 779dc302602ae8842084807ca868a491ea99b676 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
04551f17999d742c53dfb1b36286b119b448550e 
  src/test/python/apache/aurora/client/cli/test_options.py 
f2aae5741611909d5a1700cb4da70dec313e00da 
  src/test/python/apache/aurora/client/cli/test_supdate.py 
92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
  src/test/python/apache/aurora/client/cli/util.py 
aac9f9c802e4ad1f06cee8cf3f56749760b33af5 

Diff: https://reviews.apache.org/r/51384/diff/


Testing
---

./build-support/jenkins/build.sh
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
./pants test src/test/python/apache/aurora/client/cli:cli


Thanks,

Santhosh Kumar Shanmugham

Review Request 51384: Introduce UpdateMetadata fields in JobUpdateRequest.

2016-08-24 Thread Santhosh Kumar Shanmugham 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51384/
---

Review request for Aurora, David McLaughlin, Joshua Cohen, and Maxim 
Khutornenko.


Bugs: AURORA-1711
https://issues.apache.org/jira/browse/AURORA-1711


Repository: aurora


Description
---

Allow custom metadata to be stored with the job updates. This feature
is to help case the reconciliation on job update request on the clients.

Today when a client's requests are proxied via a middle-man,
if the middle-man times out before the scheduler responds with success,
the client is left the only option of retry. In the case of updates,
since multiple updates are not allowed in parallel, the retries fail,
with INVALID_REQUEST. Although the original update is already accepted
by the scheduler and is in-progress.

With this feature, clients can reconcile the job updates,
- by marking updates them with a unique id in the metadata field
- scheduler returns the in-progress job update in its response
- client can match the client-generated ids to reconcile its state.


Diffs
-

  api/src/main/thrift/org/apache/aurora/gen/api.thrift 
c5765b70501c101f0535b4eed94e9948c36808f9 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStore.java 
d2673e6b328cb1e249fbe91d18e0d9e935636eaa 
  
src/main/java/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.java
 a3b04949f1726f110638e4f93ef45947cdb9e7f8 
  src/main/java/org/apache/aurora/scheduler/storage/db/views/DbJobUpdate.java 
78703e92c932cd5e93ff0b70f2a0b6928f6b4003 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
929d021a336c6a3438613c9340c84a1096dc9069 
  
src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 
594bb6219294dcc77d48dcad14e2a6f9caa0c534 
  
src/main/java/org/apache/aurora/scheduler/updater/UpdateInprogressException.java
 PRE-CREATION 
  src/main/python/apache/aurora/client/api/__init__.py 
9149c3018ae58d405f284fcbd4076d251ccc8192 
  src/main/python/apache/aurora/client/cli/context.py 
f1a256a8d09d23d8d4d4ee7d264be0fe376398c4 
  src/main/python/apache/aurora/client/cli/options.py 
1245ff15a69a4b4347672f7b556985521e813a00 
  src/main/python/apache/aurora/client/cli/update.py 
23aaa2c1b67599420408633733e4581553f7151b 
  src/main/python/apache/aurora/client/hooks/hooked_api.py 
542f165add0f1d01a782fe6d6089bff3e736fb82 
  
src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml
 8563630a179cb6d1e3b0e22c730ccbfe1c9291e2 
  src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
a40830fd668aa650c22a48cbc757b45aef27e961 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 
08530397ff75081bde6f07f9d53317b5486e0da4 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 779dc302602ae8842084807ca868a491ea99b676 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
04551f17999d742c53dfb1b36286b119b448550e 
  src/test/python/apache/aurora/client/cli/test_options.py 
f2aae5741611909d5a1700cb4da70dec313e00da 
  src/test/python/apache/aurora/client/cli/test_supdate.py 
92fb039fb7d3e368d7c0dfa5ebdb465c7f112416 
  src/test/python/apache/aurora/client/cli/util.py 
aac9f9c802e4ad1f06cee8cf3f56749760b33af5 

Diff: https://reviews.apache.org/r/51384/diff/


Testing
---

./build-support/jenkins/build.sh
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
./pants test src/test/python/apache/aurora/client/cli:cli


Thanks,

Santhosh Kumar Shanmugham