Re: null buffers in primitive arrays

2020-11-02 Thread Wes McKinney
The ones in http://arrow.apache.org/docs/format/Versioning.html As far as the Arrow format itself is concerned, the change that you have described is not "breaking" because it was always permissible for the validity bitmap to be unallocated when the null count is 0. Previously it was handled inco

Re: null buffers in primitive arrays

2020-11-02 Thread Niranda Perera
I see. So, what are the backward compatibility guarantees Arrow has moving forward? On Mon, Nov 2, 2020 at 9:52 AM Wes McKinney wrote: > No, you'd have to follow the project's pull requests or JIRA issues, > that's the only place where these things are discussed (except > occasionally on the mai

Re: null buffers in primitive arrays

2020-11-02 Thread Wes McKinney
No, you'd have to follow the project's pull requests or JIRA issues, that's the only place where these things are discussed (except occasionally on the mailing list). On Mon, Nov 2, 2020 at 8:50 AM Niranda Perera wrote: > > Thanks Wes. Is there a document/ link for architectural decisions like >

Re: null buffers in primitive arrays

2020-11-02 Thread Niranda Perera
Thanks Wes. Is there a document/ link for architectural decisions like this? (apart from a release change log, that is) On Mon, Nov 2, 2020 at 9:26 AM Wes McKinney wrote: > Indeed, we made a change to cause buffers[0] to always be null when > the null count is 0, which has always been permitted

Re: null buffers in primitive arrays

2020-11-02 Thread Wes McKinney
Indeed, we made a change to cause buffers[0] to always be null when the null count is 0, which has always been permitted by the columnar format specification (and in 0.16.0 and prior it was inconsistently null depending on how the array was created). On Mon, Nov 2, 2020 at 8:22 AM Niranda Perera