> On March 23, 2016, 8:35 a.m., John Sirois wrote:
> > I think this change stands on its own aside from the current state of the
> > generated Go thrift bindings, but there has been a good deal of discussion
> > about those bindings offline. Some homework below.
> >
> > For the case of the `TaskQuery` thrift struct, thrift 0.9.3 generates the
> > following Go struct:
> > ```go
> > type TaskQuery struct {
> > // unused field # 1
> > JobName string `thrift:"jobName,2" json:"jobName"`
> > // unused field # 3
> > TaskIds map[string]bool `thrift:"taskIds,4" json:"taskIds"`
> > Statuses map[ScheduleStatus]bool `thrift:"statuses,5" json:"statuses"`
> > // unused field # 6
> > InstanceIds map[int32]bool `thrift:"instanceIds,7" json:"instanceIds"`
> > // unused field # 8
> > Environment string `thrift:"environment,9" json:"environment"`
> > SlaveHosts map[string]bool `thrift:"slaveHosts,10" json:"slaveHosts"`
> > JobKeys map[*JobKey]bool `thrift:"jobKeys,11" json:"jobKeys"`
> > Offset int32 `thrift:"offset,12" json:"offset"`
> > Limit int32 `thrift:"limit,13" json:"limit"`
> > Role string `thrift:"role,14" json:"role"`
> > }
> > ```
> >
> > This is reasonable since `api.TaskQuery{}.TaskIds == nil` is true; ie the
> > collections (maps represent sets here) zero to nil.
> > The issue comes in the serialization for these fields, `taskIds` is shown
> > below as an example:
> > ```go
> > func (p *TaskQuery) writeField4(oprot thrift.TProtocol) (err error) {
> > if err := oprot.WriteFieldBegin("taskIds", thrift.SET, 4); err != nil {
> > return thrift.PrependError(fmt.Sprintf("%T write field begin
> > error 4:taskIds: ", p), err)
> > }
> > if err := oprot.WriteSetBegin(thrift.STRING, len(p.TaskIds)); err !=
> > nil {
> > return thrift.PrependError("error writing set begin: ", err)
> > }
> > for v, _ := range p.TaskIds {
> > if err := oprot.WriteString(string(v)); err != nil {
> > return thrift.PrependError(fmt.Sprintf("%T. (0) field
> > write error: ", p), err)
> > }
> > }
> > if err := oprot.WriteSetEnd(); err != nil {
> > return thrift.PrependError("error writing set end: ", err)
> > }
> > if err := oprot.WriteFieldEnd(); err != nil {
> > return thrift.PrependError(fmt.Sprintf("%T write field end
> > error 4:taskIds: ", p), err)
> > }
> > return err
> > }
> > ```
> >
> > So, since its safe to do so in Go (`len(p.TaskIds) == 0` and `for v, _ :=
> > range p.TaskIds {` loops 0 times for `p.TaskIds == nil`), the code always
> > emits the `taskIds` field, whether nil or not, which presents on the other
> > end of the wire as an empty set (as opposed to a null or un-set set). This
> > does seem like a clear bug in the thrift compiler.
> > https://issues.apache.org/jira/browse/THRIFT-3700 is similar, but on the
> > deserialization side of things so I've filed
> > https://issues.apache.org/jira/browse/THRIFT-3752.
Hrm, so this may not be a thrift compiler bug per-se. Marking all TaskQuery
thrift fields as optional yields:
```go
type TaskQuery struct {
// unused field # 1
JobName *string `thrift:"jobName,2" json:"jobName,omitempty"`
// unused field # 3
TaskIds map[string]bool `thrift:"taskIds,4"
json:"taskIds,omitempty"`
Statuses map[ScheduleStatus]bool `thrift:"statuses,5"
json:"statuses,omitempty"`
// unused field # 6
InstanceIds map[int32]bool `thrift:"instanceIds,7"
json:"instanceIds,omitempty"`
// unused field # 8
Environment *string `thrift:"environment,9"
json:"environment,omitempty"`
SlaveHosts map[string]bool `thrift:"slaveHosts,10"
json:"slaveHosts,omitempty"`
JobKeys map[*JobKey]bool `thrift:"jobKeys,11"
json:"jobKeys,omitempty"`
Offset *int32 `thrift:"offset,12"
json:"offset,omitempty"`
Limit *int32 `thrift:"limit,13" json:"limit,omitempty"`
Role *string `thrift:"role,14" json:"role,omitempty"`
}
```
So map (set) fields are unchanged (still `nil`able), but primitives - not
`nil`able before, are now represented
as `nil`able pointers. This also has the effect of emitting `IsSet*` methods
and respecting these methods as a
gate for field serialization:
```go
func (p *TaskQuery) IsSetTaskIds() bool {
return p.TaskIds != nil
}
func (p *TaskQuery) writeField4(oprot thrift.TProtocol) (err error) {
if p.IsSetTaskIds() {
if err := oprot.WriteFieldBegin("taskIds", thrift.SET, 4); err
!= nil {
return thrift.PrependError(fmt.Sprintf("%T write field
begin error 4:taskIds: ", p), err)
}
if err := oprot.WriteSetBegin(thrift.STRING, len(p.TaskIds));
err != nil {
return thrift.PrependError("error writing set begin: ",
err)
}
for v, _ := range p.TaskIds {
if err := oprot.WriteString(string(v)); err != nil {
return thrift.PrependError(fmt.Sprintf("%T. (0)
field write error: ", p), err)
}
}
if err := oprot.WriteSetEnd(); err != nil {
return thrift.PrependError("error writing set end: ",
err)
}
if err := oprot.WriteFieldEnd(); err != nil {
return thrift.PrependError(fmt.Sprintf("%T write field
end error 4:taskIds: ", p), err)
}
}
return err
}
```
In other words, the thirft compiler for Go has a different notion of unset
requiredness than the java compiler - and the perils of the global lack of
specificity as to how to handle optional vs required vs <none> is well
documented :/
- John
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45193/#review125024
-----------------------------------------------------------
On March 23, 2016, 10:35 a.m., John Sirois wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45193/
> -----------------------------------------------------------
>
> (Updated March 23, 2016, 10:35 a.m.)
>
>
> Review request for Aurora, David Chung, Bill Farner, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> -------
>
> Previously, `null` was handled differently from an empty collection in
> task queries. For the Go thrift bindings, this was problematic since
> zero values in Go are useful in almost all cases and in particular in the
> case of maps (used to represent sets). In these cases unset `TaskQuery`
> collection parameters are serialized as empty collections (empty
> maps) instead of `nil` (`null`), leading to the inability to use the
> query API in any natural way.
>
> src/main/java/org/apache/aurora/scheduler/base/JobKeys.java
> | 2 +-
> src/main/java/org/apache/aurora/scheduler/base/Query.java
> | 2 +-
> src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java
> | 2 +-
> src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java
> | 6 ++++--
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml
> | 6 +++---
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java
> | 20 +++++++++++++++++---
> 6 files changed, 27 insertions(+), 11 deletions(-)
>
>
> Diffs
> -----
>
> src/main/java/org/apache/aurora/scheduler/base/JobKeys.java
> 8f5bf58b963ae5f76aad7dfa34bae5b9e67d6242
> src/main/java/org/apache/aurora/scheduler/base/Query.java
> ee01eaa4d0230d6bf0909b6460f27a74f03240db
> src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java
> ac0bb374842741d7ccb7a83c574a90ac156af0f9
> src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java
> 231a55615abfbb483667f5f8ef71d2709fc16a88
> src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml
> 684614ffc42dd6778c7675a6c2f81cb72c106c0e
>
> src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java
> e56fed2e6c0cdb47737cf1a9b637c44c5e5b9815
>
> Diff: https://reviews.apache.org/r/45193/diff/
>
>
> Testing
> -------
>
> NB: This change was broken out of https://reviews.apache.org/r/42756/
> since it stands on its own (although its slightly more awkward in the
> mutable thrift world) and the case of the Go Aurora API client forces the
> issue.
>
> Locally green:
> ```
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> ```
>
>
> Thanks,
>
> John Sirois
>
>
