Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-28 Thread Bill Farner

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125678
---


Ship it!




- Bill Farner


On March 28, 2016, 8:33 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 28, 2016, 8:33 a.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets).  In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> 078dd8b63fdca192c735f9097edd030ee315a021 
>   src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 
> 4bf40047e105389ac7139edc449857889d390106 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> 231a55615abfbb483667f5f8ef71d2709fc16a88 
>   src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
> d326d24dd527d084bce1b300f1818d3b1d94c036 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  5d246bee1a4dabc563a23c542384205537719f6a 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e 
>   
> src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java
>  dfe94d3fadc3f5e3322dd5a3a367ad6ef22c2a99 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  3ba03429748448642571cfe0858278a50148745a 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0a7b518578f4fd62c22e3ba52d8beae7958dc9eb 
> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> ---
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John Sirois
> 
>



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-28 Thread Aurora ReviewBot

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125657
---


Ship it!




Master (83a078b) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On March 28, 2016, 3:33 p.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 28, 2016, 3:33 p.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets).  In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> 078dd8b63fdca192c735f9097edd030ee315a021 
>   src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 
> 4bf40047e105389ac7139edc449857889d390106 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> 231a55615abfbb483667f5f8ef71d2709fc16a88 
>   src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
> d326d24dd527d084bce1b300f1818d3b1d94c036 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  5d246bee1a4dabc563a23c542384205537719f6a 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e 
>   
> src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java
>  dfe94d3fadc3f5e3322dd5a3a367ad6ef22c2a99 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  3ba03429748448642571cfe0858278a50148745a 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0a7b518578f4fd62c22e3ba52d8beae7958dc9eb 
> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> ---
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John Sirois
> 
>



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-28 Thread John Sirois

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125644
---



Bill - I'll take silence as consent and merge this today.

- John Sirois


On March 28, 2016, 9:33 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 28, 2016, 9:33 a.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets).  In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> 078dd8b63fdca192c735f9097edd030ee315a021 
>   src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 
> 4bf40047e105389ac7139edc449857889d390106 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> 231a55615abfbb483667f5f8ef71d2709fc16a88 
>   src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
> d326d24dd527d084bce1b300f1818d3b1d94c036 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  5d246bee1a4dabc563a23c542384205537719f6a 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e 
>   
> src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java
>  dfe94d3fadc3f5e3322dd5a3a367ad6ef22c2a99 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  3ba03429748448642571cfe0858278a50148745a 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0a7b518578f4fd62c22e3ba52d8beae7958dc9eb 
> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> ---
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John Sirois
> 
>



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-28 Thread John Sirois

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/
---

(Updated March 28, 2016, 9:33 a.m.)


Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.


Changes
---

Back task queries with `ITaskQuery`.

This avoids special null checks for collection query criteria although
it does require more care in struct handling code where `TaskQuery` <->
`ITaskQuery` are lossy for `isSet` query methods.

 src/main/java/org/apache/aurora/scheduler/base/JobKeys.java
|  8 +++-
 src/main/java/org/apache/aurora/scheduler/base/Query.java  
| 19 +--
 src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java   
| 15 +++
 src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java  
|  2 +-
 src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java   
|  4 ++--
 src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java
| 14 +++---
 src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java
| 12 
 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
|  2 +-
 src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml   
| 10 +-
 
src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java 
   | 18 ++
 
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
   |  6 +++---
 
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 | 20 ++--
 12 files changed, 66 insertions(+), 64 deletions(-)


Repository: aurora


Description
---

Previously, `null` was handled differently from an empty collection in
task queries. For the Go thrift bindings, this was problematic since
zero values in Go are useful in almost all cases and in particular in the
case of maps (used to represent sets).  In these cases unset `TaskQuery`
collection parameters are serialized as empty collections (empty
maps) instead of `nil` (`null`), leading to the inability to use the
query API in any natural way.

 src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  | 
 2 +-
 src/main/java/org/apache/aurora/scheduler/base/Query.java| 
 2 +-
 src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java | 
 2 +-
 src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  | 
 6 --
 src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml | 
 6 +++---
 src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java | 
20 +---
 6 files changed, 27 insertions(+), 11 deletions(-)


Diffs (updated)
-

  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
  src/main/java/org/apache/aurora/scheduler/base/Query.java 
ee01eaa4d0230d6bf0909b6460f27a74f03240db 
  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
ac0bb374842741d7ccb7a83c574a90ac156af0f9 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
078dd8b63fdca192c735f9097edd030ee315a021 
  src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 
4bf40047e105389ac7139edc449857889d390106 
  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
231a55615abfbb483667f5f8ef71d2709fc16a88 
  src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
d326d24dd527d084bce1b300f1818d3b1d94c036 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
5d246bee1a4dabc563a23c542384205537719f6a 
  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
684614ffc42dd6778c7675a6c2f81cb72c106c0e 
  
src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java 
dfe94d3fadc3f5e3322dd5a3a367ad6ef22c2a99 
  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
3ba03429748448642571cfe0858278a50148745a 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 0a7b518578f4fd62c22e3ba52d8beae7958dc9eb 

Diff: https://reviews.apache.org/r/45193/diff/


Testing
---

NB: This change was broken out of https://reviews.apache.org/r/42756/
since it stands on its own (although its slightly more awkward in the
mutable thrift world) and the case of the Go Aurora API client forces the
issue.

Locally green:
```
./build-support/jenkins/build.sh
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
```


Thanks,

John Sirois



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-28 Thread John Sirois

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125643
---



Rebasing to pick up https://reviews.apache.org/r/45366/

- John Sirois


On March 24, 2016, 8:52 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 24, 2016, 8:52 a.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets).  In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> 078dd8b63fdca192c735f9097edd030ee315a021 
>   src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 
> 4bf40047e105389ac7139edc449857889d390106 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> 231a55615abfbb483667f5f8ef71d2709fc16a88 
>   src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
> d326d24dd527d084bce1b300f1818d3b1d94c036 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  5d246bee1a4dabc563a23c542384205537719f6a 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e 
>   
> src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java
>  dfe94d3fadc3f5e3322dd5a3a367ad6ef22c2a99 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  3ba03429748448642571cfe0858278a50148745a 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0a7b518578f4fd62c22e3ba52d8beae7958dc9eb 
> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> ---
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John Sirois
> 
>



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-24 Thread John Sirois

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125271
---



Bill - this has changed enough that you should re-assess your ship.

- John Sirois


On March 24, 2016, 8:52 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 24, 2016, 8:52 a.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets).  In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> 078dd8b63fdca192c735f9097edd030ee315a021 
>   src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 
> 4bf40047e105389ac7139edc449857889d390106 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> 231a55615abfbb483667f5f8ef71d2709fc16a88 
>   src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
> d326d24dd527d084bce1b300f1818d3b1d94c036 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  5d246bee1a4dabc563a23c542384205537719f6a 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e 
>   
> src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java
>  dfe94d3fadc3f5e3322dd5a3a367ad6ef22c2a99 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  3ba03429748448642571cfe0858278a50148745a 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0a7b518578f4fd62c22e3ba52d8beae7958dc9eb 
> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> ---
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John Sirois
> 
>



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-24 Thread Aurora ReviewBot

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125270
---


Ship it!




Master (9927231) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On March 24, 2016, 2:52 p.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 24, 2016, 2:52 p.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets).  In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> 078dd8b63fdca192c735f9097edd030ee315a021 
>   src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 
> 4bf40047e105389ac7139edc449857889d390106 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> 231a55615abfbb483667f5f8ef71d2709fc16a88 
>   src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
> d326d24dd527d084bce1b300f1818d3b1d94c036 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  5d246bee1a4dabc563a23c542384205537719f6a 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e 
>   
> src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java
>  dfe94d3fadc3f5e3322dd5a3a367ad6ef22c2a99 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  3ba03429748448642571cfe0858278a50148745a 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0a7b518578f4fd62c22e3ba52d8beae7958dc9eb 
> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> ---
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John Sirois
> 
>



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-24 Thread John Sirois

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125265
---



@ReviewBot retry

- John Sirois


On March 24, 2016, 8:52 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 24, 2016, 8:52 a.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets).  In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> 078dd8b63fdca192c735f9097edd030ee315a021 
>   src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 
> 4bf40047e105389ac7139edc449857889d390106 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> 231a55615abfbb483667f5f8ef71d2709fc16a88 
>   src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
> d326d24dd527d084bce1b300f1818d3b1d94c036 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  5d246bee1a4dabc563a23c542384205537719f6a 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e 
>   
> src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java
>  dfe94d3fadc3f5e3322dd5a3a367ad6ef22c2a99 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  3ba03429748448642571cfe0858278a50148745a 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0a7b518578f4fd62c22e3ba52d8beae7958dc9eb 
> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> ---
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John Sirois
> 
>



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-24 Thread Zameer Manji

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125262
---


Ship it!




The ripples into the scheduler code was a little bit more than I expected, but 
I think this is the better solution. I think whenever we get around to fixing 
AURORA-1650 and AURORA-1651 the code should end up a lot cleaner.

- Zameer Manji


On March 24, 2016, 7:52 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 24, 2016, 7:52 a.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets).  In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> 078dd8b63fdca192c735f9097edd030ee315a021 
>   src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 
> 4bf40047e105389ac7139edc449857889d390106 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> 231a55615abfbb483667f5f8ef71d2709fc16a88 
>   src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
> d326d24dd527d084bce1b300f1818d3b1d94c036 
>   
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
>  5d246bee1a4dabc563a23c542384205537719f6a 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e 
>   
> src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java
>  dfe94d3fadc3f5e3322dd5a3a367ad6ef22c2a99 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java
>  3ba03429748448642571cfe0858278a50148745a 
>   
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
>  0a7b518578f4fd62c22e3ba52d8beae7958dc9eb 
> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> ---
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John Sirois
> 
>



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-24 Thread Aurora ReviewBot

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125261
---



Master (9927231) is red with this patch.
  ./build-support/jenkins/build.sh

   
self._clock.converge(threads=[hct.threaded_health_checker])
   
self._clock.assert_waiting(hct.threaded_health_checker, amount=1)
 
   assert hct._total_latency == 0
   assert 
hct.metrics.sample()['total_latency_secs'] == 0
 
   # start the health check (during health check it 
is still 0)
   epsilon = 0.001
   self._clock.tick(1.0 + epsilon)
   
self._clock.converge(threads=[hct.threaded_health_checker])
   
self._clock.assert_waiting(hct.threaded_health_checker, amount=0.5)
   assert hct._total_latency == 0
   assert 
hct.metrics.sample()['total_latency_secs'] == 0
   assert hct.metrics.sample()['checks'] == 0
 
   # finish the health check
   self._clock.tick(0.5 + epsilon)
   
self._clock.converge(threads=[hct.threaded_health_checker])
   
self._clock.assert_waiting(hct.threaded_health_checker, amount=1)  # 
interval_secs
 > assert hct._total_latency == 0.5
 E AssertionError: assert 0.5009 
== 0.5
 E  +  where 0.5009 = 
._total_latency
 
 
src/test/python/apache/aurora/executor/common/test_health_checker.py:174: 
AssertionError
 -- Captured stderr call --
 [] Time now: 0.0
 [] Time now: 0.0
 [] Time now: 1.0
 [] Time now: 1.001
 [] Time now: 1.001
 [] Time now: 1.501
 [] Time now: 1.502
  generated xml file: 
/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/test-results/415337499eb72578eab327a6487c1f5c9452b3d6.xml
 
  1 failed, 663 passed, 5 skipped, 1 warnings in 
313.43 seconds 
 
FAILURE


15:26:21 07:04   [complete]
   FAILURE


I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On March 24, 2016, 2:52 p.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 24, 2016, 2:52 p.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets).  In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
> 078dd8b63fdca192c735f9097edd030ee315a021 
>   

Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-24 Thread John Sirois


> On March 23, 2016, 1:36 p.m., Zameer Manji wrote:
> > I am in favor of making this change. However, it seems this patch could be
> > improved because the storage layer has to now check for both `null` and
> > empty collection.
> > 
> > I think a better solution would be to change Query.Builder to be a Supplier 
> > of
> > `ITaskQuery` instead of `TaskQuery`. The `I*` classes always return empty
> > collections and never null. By doing this the storage layer can be 
> > simplified to
> > only check for empty collections. This is also more future proof because the
> > storage/query layer can never be given `null` which means the next time we 
> > add a
> > new field to `TaskQuery` (such as image id?) we don't have to risk 
> > regressing on
> > this behaviour.
> > 
> > I have scanned the code and it doesn't seem to be a lot of work, it seems 
> > mostly
> > places where there are entries like `query.getStatusesSize()` would need to 
> > be
> > replaced with `query.getStatus().size()`.
> > 
> > I don't feel super strong about my suggestion here so if you do object or 
> > if it
> > does seem like a lot of work, I will just ship this change as is.
> 
> John Sirois wrote:
> I'll give it a spike and see how it works out.  I will note though that 
> the storage layer (db) - was already doing this for 2/5 collection fields in 
> the TaskMapper.select.  IOW: the scope here is expanding from making things 
> consistent to also paying down more fundamental existing debt.  I'm all for 
> that, but I also want to call out I tried paying down that debt and then some 
> by eliminating I* alltogether and using a single consistent immutable model 
> where no collections were null, but instead empty.  So this is make it nice - 
> but in a middling way.

The `TaskMapper.xml` does of course get cleaner, but the scheduler code gets 
trickier due to some I* isSet issues 
(https://issues.apache.org/jira/browse/AURORA-1650 
https://issues.apache.org/jira/browse/AURORA-1651).
See what you think.


- John


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125102
---


On March 23, 2016, 10:35 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 23, 2016, 10:35 a.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets).  In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> 231a55615abfbb483667f5f8ef71d2709fc16a88 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> ---
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John 

Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-24 Thread John Sirois

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/
---

(Updated March 24, 2016, 8:52 a.m.)


Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.


Changes
---

Back task queries with `ITaskQuery`.

This avoids special null checks for collection query criteria although
it does require more care in struct handling code where `TaskQuery` <->
`ITaskQuery` are lossy for `isSet` query methods.

 src/main/java/org/apache/aurora/scheduler/base/JobKeys.java
|  8 +++-
 src/main/java/org/apache/aurora/scheduler/base/Query.java  
| 19 +--
 src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java   
| 15 +++
 src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java  
|  2 +-
 src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java   
|  4 ++--
 src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java
| 14 +++---
 src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java
| 12 
 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
|  2 +-
 src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml   
| 10 +-
 
src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java 
   | 18 ++
 
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
   |  6 +++---
 
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 | 20 ++--
 12 files changed, 66 insertions(+), 64 deletions(-)


Repository: aurora


Description
---

Previously, `null` was handled differently from an empty collection in
task queries. For the Go thrift bindings, this was problematic since
zero values in Go are useful in almost all cases and in particular in the
case of maps (used to represent sets).  In these cases unset `TaskQuery`
collection parameters are serialized as empty collections (empty
maps) instead of `nil` (`null`), leading to the inability to use the
query API in any natural way.

 src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  | 
 2 +-
 src/main/java/org/apache/aurora/scheduler/base/Query.java| 
 2 +-
 src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java | 
 2 +-
 src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  | 
 6 --
 src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml | 
 6 +++---
 src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java | 
20 +---
 6 files changed, 27 insertions(+), 11 deletions(-)


Diffs (updated)
-

  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
  src/main/java/org/apache/aurora/scheduler/base/Query.java 
ee01eaa4d0230d6bf0909b6460f27a74f03240db 
  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
ac0bb374842741d7ccb7a83c574a90ac156af0f9 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 
078dd8b63fdca192c735f9097edd030ee315a021 
  src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java 
4bf40047e105389ac7139edc449857889d390106 
  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
231a55615abfbb483667f5f8ef71d2709fc16a88 
  src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 
d326d24dd527d084bce1b300f1818d3b1d94c036 
  
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 
5d246bee1a4dabc563a23c542384205537719f6a 
  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
684614ffc42dd6778c7675a6c2f81cb72c106c0e 
  
src/test/java/org/apache/aurora/scheduler/http/api/security/HttpSecurityIT.java 
dfe94d3fadc3f5e3322dd5a3a367ad6ef22c2a99 
  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
  
src/test/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImplTest.java 
3ba03429748448642571cfe0858278a50148745a 
  
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
 0a7b518578f4fd62c22e3ba52d8beae7958dc9eb 

Diff: https://reviews.apache.org/r/45193/diff/


Testing
---

NB: This change was broken out of https://reviews.apache.org/r/42756/
since it stands on its own (although its slightly more awkward in the
mutable thrift world) and the case of the Go Aurora API client forces the
issue.

Locally green:
```
./build-support/jenkins/build.sh
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
```


Thanks,

John Sirois



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-23 Thread John Sirois


> On March 23, 2016, 1:36 p.m., Zameer Manji wrote:
> > I am in favor of making this change. However, it seems this patch could be
> > improved because the storage layer has to now check for both `null` and
> > empty collection.
> > 
> > I think a better solution would be to change Query.Builder to be a Supplier 
> > of
> > `ITaskQuery` instead of `TaskQuery`. The `I*` classes always return empty
> > collections and never null. By doing this the storage layer can be 
> > simplified to
> > only check for empty collections. This is also more future proof because the
> > storage/query layer can never be given `null` which means the next time we 
> > add a
> > new field to `TaskQuery` (such as image id?) we don't have to risk 
> > regressing on
> > this behaviour.
> > 
> > I have scanned the code and it doesn't seem to be a lot of work, it seems 
> > mostly
> > places where there are entries like `query.getStatusesSize()` would need to 
> > be
> > replaced with `query.getStatus().size()`.
> > 
> > I don't feel super strong about my suggestion here so if you do object or 
> > if it
> > does seem like a lot of work, I will just ship this change as is.

I'll give it a spike and see how it works out.  I will note though that the 
storage layer (db) - was already doing this for 2/5 collection fields in the 
TaskMapper.select.  IOW: the scope here is expanding from making things 
consistent to also paying down more fundamental existing debt.  I'm all for 
that, but I also want to call out I tried paying down that debt and then some 
by eliminating I* alltogether and using a single consistent immutable model 
where no collections were null, but instead empty.  So this is make it nice - 
but in a middling way.


- John


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125102
---


On March 23, 2016, 10:35 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 23, 2016, 10:35 a.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets).  In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> 231a55615abfbb483667f5f8ef71d2709fc16a88 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> ---
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John Sirois
> 
>



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-23 Thread Zameer Manji

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125102
---



I am in favor of making this change. However, it seems this patch could be
improved because the storage layer has to now check for both `null` and
empty collection.

I think a better solution would be to change Query.Builder to be a Supplier of
`ITaskQuery` instead of `TaskQuery`. The `I*` classes always return empty
collections and never null. By doing this the storage layer can be simplified to
only check for empty collections. This is also more future proof because the
storage/query layer can never be given `null` which means the next time we add a
new field to `TaskQuery` (such as image id?) we don't have to risk regressing on
this behaviour.

I have scanned the code and it doesn't seem to be a lot of work, it seems mostly
places where there are entries like `query.getStatusesSize()` would need to be
replaced with `query.getStatus().size()`.

I don't feel super strong about my suggestion here so if you do object or if it
does seem like a lot of work, I will just ship this change as is.

- Zameer Manji


On March 23, 2016, 9:35 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 23, 2016, 9:35 a.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets).  In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> 231a55615abfbb483667f5f8ef71d2709fc16a88 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> ---
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John Sirois
> 
>



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-23 Thread John Sirois


> On March 23, 2016, 8:35 a.m., John Sirois wrote:
> > I think this change stands on its own aside from the current state of the 
> > generated Go thrift bindings, but there has been a good deal of discussion 
> > about those bindings offline.  Some homework below.
> > 
> > For the case of the `TaskQuery` thrift struct, thrift 0.9.3 generates the 
> > following Go struct:
> > ```go
> > type TaskQuery struct {
> > // unused field # 1
> > JobName string `thrift:"jobName,2" json:"jobName"`
> > // unused field # 3
> > TaskIds  map[string]bool `thrift:"taskIds,4" json:"taskIds"`
> > Statuses map[ScheduleStatus]bool `thrift:"statuses,5" json:"statuses"`
> > // unused field # 6
> > InstanceIds map[int32]bool `thrift:"instanceIds,7" json:"instanceIds"`
> > // unused field # 8
> > Environment string   `thrift:"environment,9" json:"environment"`
> > SlaveHosts  map[string]bool  `thrift:"slaveHosts,10" json:"slaveHosts"`
> > JobKeys map[*JobKey]bool `thrift:"jobKeys,11" json:"jobKeys"`
> > Offset  int32`thrift:"offset,12" json:"offset"`
> > Limit   int32`thrift:"limit,13" json:"limit"`
> > Rolestring   `thrift:"role,14" json:"role"`
> > }
> > ```
> > 
> > This is reasonable since `api.TaskQuery{}.TaskIds == nil` is true; ie the 
> > collections (maps represent sets here) zero to nil.
> > The issue comes in the serialization for these fields, `taskIds` is shown 
> > below as an example:
> > ```go
> > func (p *TaskQuery) writeField4(oprot thrift.TProtocol) (err error) {
> > if err := oprot.WriteFieldBegin("taskIds", thrift.SET, 4); err != nil {
> > return thrift.PrependError(fmt.Sprintf("%T write field begin 
> > error 4:taskIds: ", p), err)
> > }
> > if err := oprot.WriteSetBegin(thrift.STRING, len(p.TaskIds)); err != 
> > nil {
> > return thrift.PrependError("error writing set begin: ", err)
> > }
> > for v, _ := range p.TaskIds {
> > if err := oprot.WriteString(string(v)); err != nil {
> > return thrift.PrependError(fmt.Sprintf("%T. (0) field 
> > write error: ", p), err)
> > }
> > }
> > if err := oprot.WriteSetEnd(); err != nil {
> > return thrift.PrependError("error writing set end: ", err)
> > }
> > if err := oprot.WriteFieldEnd(); err != nil {
> > return thrift.PrependError(fmt.Sprintf("%T write field end 
> > error 4:taskIds: ", p), err)
> > }
> > return err
> > }
> > ```
> > 
> > So, since its safe to do so in Go (`len(p.TaskIds) == 0` and `for v, _ := 
> > range p.TaskIds {` loops 0 times for `p.TaskIds == nil`), the code always 
> > emits the `taskIds` field, whether nil or not, which presents on the other 
> > end of the wire as an empty set (as opposed to a null or un-set set).  This 
> > does seem like a clear bug in the thrift compiler.  
> > https://issues.apache.org/jira/browse/THRIFT-3700 is similar, but on the 
> > deserialization side of things so I've filed 
> > https://issues.apache.org/jira/browse/THRIFT-3752.

Hrm, so this may not be a thrift compiler bug per-se.  Marking all TaskQuery 
thrift fields as optional yields:
```go
type TaskQuery struct {
// unused field # 1
JobName *string `thrift:"jobName,2" json:"jobName,omitempty"`
// unused field # 3
TaskIds  map[string]bool `thrift:"taskIds,4" 
json:"taskIds,omitempty"`
Statuses map[ScheduleStatus]bool `thrift:"statuses,5" 
json:"statuses,omitempty"`
// unused field # 6
InstanceIds map[int32]bool `thrift:"instanceIds,7" 
json:"instanceIds,omitempty"`
// unused field # 8
Environment *string  `thrift:"environment,9" 
json:"environment,omitempty"`
SlaveHosts  map[string]bool  `thrift:"slaveHosts,10" 
json:"slaveHosts,omitempty"`
JobKeys map[*JobKey]bool `thrift:"jobKeys,11" 
json:"jobKeys,omitempty"`
Offset  *int32   `thrift:"offset,12" 
json:"offset,omitempty"`
Limit   *int32   `thrift:"limit,13" json:"limit,omitempty"`
Role*string  `thrift:"role,14" json:"role,omitempty"`
}
```

So map (set) fields are unchanged (still `nil`able), but primitives - not 
`nil`able before, are now represented
as `nil`able pointers.  This also has the effect of emitting `IsSet*` methods 
and respecting these methods as a
gate for field serialization:
```go
func (p *TaskQuery) IsSetTaskIds() bool {
return p.TaskIds != nil
}

func (p *TaskQuery) writeField4(oprot thrift.TProtocol) (err error) {
if p.IsSetTaskIds() {
if err := oprot.WriteFieldBegin("taskIds", thrift.SET, 4); err 
!= nil {
return thrift.PrependError(fmt.Sprintf("%T write field 
begin error 4:taskIds: ", p), err)
}
if err := oprot.WriteSetBegin(thrift.STRING, len(p.TaskIds)); 
err != nil {
   

Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-23 Thread John Sirois

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/
---

(Updated March 23, 2016, 10:35 a.m.)


Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.


Changes
---

Update the description to reflect homework.


Repository: aurora


Description (updated)
---

Previously, `null` was handled differently from an empty collection in
task queries. For the Go thrift bindings, this was problematic since
zero values in Go are useful in almost all cases and in particular in the
case of maps (used to represent sets).  In these cases unset `TaskQuery`
collection parameters are serialized as empty collections (empty
maps) instead of `nil` (`null`), leading to the inability to use the
query API in any natural way.

 src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  | 
 2 +-
 src/main/java/org/apache/aurora/scheduler/base/Query.java| 
 2 +-
 src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java | 
 2 +-
 src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  | 
 6 --
 src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml | 
 6 +++---
 src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java | 
20 +---
 6 files changed, 27 insertions(+), 11 deletions(-)


Diffs
-

  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
  src/main/java/org/apache/aurora/scheduler/base/Query.java 
ee01eaa4d0230d6bf0909b6460f27a74f03240db 
  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
ac0bb374842741d7ccb7a83c574a90ac156af0f9 
  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
231a55615abfbb483667f5f8ef71d2709fc16a88 
  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
684614ffc42dd6778c7675a6c2f81cb72c106c0e 
  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 

Diff: https://reviews.apache.org/r/45193/diff/


Testing
---

NB: This change was broken out of https://reviews.apache.org/r/42756/
since it stands on its own (although its slightly more awkward in the
mutable thrift world) and the case of the Go Aurora API client forces the
issue.

Locally green:
```
./build-support/jenkins/build.sh
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
```


Thanks,

John Sirois



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-23 Thread John Sirois

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125024
---



I think this change stands on its own aside from the current state of the 
generated Go thrift bindings, but there has been a good deal of discussion 
about those bindings offline.  Some homework below.

For the case of the `TaskQuery` thrift struct, thrift 0.9.3 generates the 
following Go struct:
```go
type TaskQuery struct {
// unused field # 1
JobName string `thrift:"jobName,2" json:"jobName"`
// unused field # 3
TaskIds  map[string]bool `thrift:"taskIds,4" json:"taskIds"`
Statuses map[ScheduleStatus]bool `thrift:"statuses,5" json:"statuses"`
// unused field # 6
InstanceIds map[int32]bool `thrift:"instanceIds,7" json:"instanceIds"`
// unused field # 8
Environment string   `thrift:"environment,9" json:"environment"`
SlaveHosts  map[string]bool  `thrift:"slaveHosts,10" json:"slaveHosts"`
JobKeys map[*JobKey]bool `thrift:"jobKeys,11" json:"jobKeys"`
Offset  int32`thrift:"offset,12" json:"offset"`
Limit   int32`thrift:"limit,13" json:"limit"`
Rolestring   `thrift:"role,14" json:"role"`
}
```

This is reasonable since `api.TaskQuery{}.TaskIds == nil` is true; ie the 
collections (maps represent sets here) zero to nil.
The issue comes in the serialization for these fields, `taskIds` is shown below 
as an example:
```go
func (p *TaskQuery) writeField4(oprot thrift.TProtocol) (err error) {
if err := oprot.WriteFieldBegin("taskIds", thrift.SET, 4); err != nil {
return thrift.PrependError(fmt.Sprintf("%T write field begin 
error 4:taskIds: ", p), err)
}
if err := oprot.WriteSetBegin(thrift.STRING, len(p.TaskIds)); err != 
nil {
return thrift.PrependError("error writing set begin: ", err)
}
for v, _ := range p.TaskIds {
if err := oprot.WriteString(string(v)); err != nil {
return thrift.PrependError(fmt.Sprintf("%T. (0) field 
write error: ", p), err)
}
}
if err := oprot.WriteSetEnd(); err != nil {
return thrift.PrependError("error writing set end: ", err)
}
if err := oprot.WriteFieldEnd(); err != nil {
return thrift.PrependError(fmt.Sprintf("%T write field end 
error 4:taskIds: ", p), err)
}
return err
}
```

So, since its safe to do so in Go (`len(p.TaskIds) == 0` and `for v, _ := range 
p.TaskIds {` loops 0 times for `p.TaskIds == nil`), the code always emits the 
`taskIds` field, whether nil or not, which presents on the other end of the 
wire as an empty set (as opposed to a null or un-set set).  This does seem like 
a clear bug in the thrift compiler.  
https://issues.apache.org/jira/browse/THRIFT-3700 is similar, but on the 
deserialization side of things so I've filed 
https://issues.apache.org/jira/browse/THRIFT-3752.

- John Sirois


On March 22, 2016, 8:32 p.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 22, 2016, 8:32 p.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of slices (collections).  In these cases unset `TaskQuery`
> collection parameters are represented as empty collections (empty
> slices[]) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   

Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-22 Thread Bill Farner

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review124954
---


Ship it!




Ship It!

- Bill Farner


On March 22, 2016, 7:32 p.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 22, 2016, 7:32 p.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of slices (collections).  In these cases unset `TaskQuery`
> collection parameters are represented as empty collections (empty
> slices[]) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> 231a55615abfbb483667f5f8ef71d2709fc16a88 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> ---
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John Sirois
> 
>



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-22 Thread Aurora ReviewBot

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review124951
---


Ship it!




Master (c66a9ee) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On March 23, 2016, 2:32 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> ---
> 
> (Updated March 23, 2016, 2:32 a.m.)
> 
> 
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of slices (collections).  In these cases unset `TaskQuery`
> collection parameters are represented as empty collections (empty
> slices[]) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
> 
>  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/base/Query.java
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> |  2 +-
>  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  
> |  6 --
>  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> |  6 +++---
>  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> | 20 +---
>  6 files changed, 27 insertions(+), 11 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
>   src/main/java/org/apache/aurora/scheduler/base/Query.java 
> ee01eaa4d0230d6bf0909b6460f27a74f03240db 
>   src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
> ac0bb374842741d7ccb7a83c574a90ac156af0f9 
>   src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
> 231a55615abfbb483667f5f8ef71d2709fc16a88 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e 
>   
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 
> 
> Diff: https://reviews.apache.org/r/45193/diff/
> 
> 
> Testing
> ---
> 
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
> 
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
> 
> 
> Thanks,
> 
> John Sirois
> 
>



Re: Review Request 45193: Treat empty and null collections equivalently in task queries.

2016-03-22 Thread John Sirois

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/
---

(Updated March 22, 2016, 8:32 p.m.)


Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.


Repository: aurora


Description (updated)
---

Previously, `null` was handled differently from an empty collection in
task queries. For the Go thrift bindings, this was problematic since
zero values in Go are useful in almost all cases and in particular in the
case of slices (collections).  In these cases unset `TaskQuery`
collection parameters are represented as empty collections (empty
slices[]) instead of `nil` (`null`), leading to the inability to use the
query API in any natural way.

 src/main/java/org/apache/aurora/scheduler/base/JobKeys.java  | 
 2 +-
 src/main/java/org/apache/aurora/scheduler/base/Query.java| 
 2 +-
 src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java | 
 2 +-
 src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java  | 
 6 --
 src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml | 
 6 +++---
 src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java | 
20 +---
 6 files changed, 27 insertions(+), 11 deletions(-)


Diffs
-

  src/main/java/org/apache/aurora/scheduler/base/JobKeys.java 
8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242 
  src/main/java/org/apache/aurora/scheduler/base/Query.java 
ee01eaa4d0230d6bf0909b6460f27a74f03240db 
  src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java 
ac0bb374842741d7ccb7a83c574a90ac156af0f9 
  src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 
231a55615abfbb483667f5f8ef71d2709fc16a88 
  src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml 
684614ffc42dd6778c7675a6c2f81cb72c106c0e 
  src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 
e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815 

Diff: https://reviews.apache.org/r/45193/diff/


Testing
---

NB: This change was broken out of https://reviews.apache.org/r/42756/
since it stands on its own (although its slightly more awkward in the
mutable thrift world) and the case of the Go Aurora API client forces the
issue.

Locally green:
```
./build-support/jenkins/build.sh
./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
```


Thanks,

John Sirois