Re: Sparse Union format

2020-05-19 Thread Micah Kornfield
Hi Ryan, In addition to the limitations mentioned above another one is only 1 column of each type that can participate in the union. There are some old threads on these differences on the mailing list that should be searchable. Thanks, Micah On Tue, May 19, 2020 at 6:44 AM Antoine Pitrou

Re: Sparse Union format

2020-05-19 Thread Antoine Pitrou
Also, you may want to run the integration tests and inspect the generated JSON file for union data, it will probably be informative (look for type ids). Regards Antoine. Le 19/05/2020 à 15:38, Ryan Murray a écrit : > Thanks for the clarification! Next time I will read the whole document ;-)

Re: Sparse Union format

2020-05-19 Thread Ryan Murray
Thanks for the clarification! Next time I will read the whole document ;-) On Tue, May 19, 2020 at 2:38 PM Antoine Pitrou wrote: > > As explained in the comment below: > https://github.com/apache/arrow/blob/master/format/Schema.fbs#L91 > > Regards > > Antoine. > > > Le 19/05/2020 à 14:14, Ryan

Re: Sparse Union format

2020-05-19 Thread Antoine Pitrou
As explained in the comment below: https://github.com/apache/arrow/blob/master/format/Schema.fbs#L91 Regards Antoine. Le 19/05/2020 à 14:14, Ryan Murray a écrit : > Thanks Antoine, > > Can you just clarify what you mean by 'type ids are logical'? In my mind > type ids are strongly coupled

Re: Sparse Union format

2020-05-19 Thread Ryan Murray
Thanks Antoine, Can you just clarify what you mean by 'type ids are logical'? In my mind type ids are strongly coupled to the types and their order in Schema.fbs [1]. Do you mean that the order there is only a convention and we can't assume that 0 === Null? Best, Ryan [1]

Re: Sparse Union format

2020-05-19 Thread Antoine Pitrou
Le 19/05/2020 à 13:43, Ryan Murray a écrit : > Hey All, > > While working on https://issues.apache.org/jira/browse/ARROW-1692 I noticed > that there is a difference between C++ and Java on the way Sparse Unions > are handled. I haven't seen in the format spec which the correct is so I > wanted

Sparse Union format

2020-05-19 Thread Ryan Murray
Hey All, While working on https://issues.apache.org/jira/browse/ARROW-1692 I noticed that there is a difference between C++ and Java on the way Sparse Unions are handled. I haven't seen in the format spec which the correct is so I wanted to check with the wider community. c++ (and the