+1 (binding) (I had a couple of nits on language, that I put in the PR On Mon, Jun 29, 2020 at 2:24 PM Wes McKinney <wesmck...@gmail.com> wrote:
> Hi, > > As discussed on the mailing list [1], it has been proposed to remove > the validity bitmap buffer from Union types in the columnar format > specification and instead let value validity be determined exclusively > by constituent arrays of the union. > > One of the primary motivations for this is to simplify the creation of > unions, since constructing a validity bitmap that merges the > information contained in the child arrays' bitmaps is quite > complicated. > > Note that change breaks IPC forward compatibility for union types, > however implementations with hitherto spec-compliant union > implementations would be able to (at their discretion, of course) > preserve backward compatibility for deserializing "old" union data in > the case that the parent null count of the union is zero. The expected > impact of this breakage is low, particularly given that Unions have > been absent from integration testing and thus not recommended for > anything but ephemeral serialization. > > Under the assumption that the MetadataVersion V4 -> V5 version bump is > accepted, in order to protect against forward compatibility problems, > Arrow implementations would be forbidden from serializing union types > using the MetadataVersion::V4. > > A PR with the changes to Columnar.rst is at [2]. > > The vote will be open for at least 72 hours. > > [ ] +1 Accept changes to Columnar.rst (removing union validity bitmaps) > [ ] +0 > [ ] -1 Do not accept changes because... > > [1]: > https://lists.apache.org/thread.html/r889d7532cf1e1eff74b072b4e642762ad39f4008caccef5ecde5b26e%40%3Cdev.arrow.apache.org%3E > [2]: https://github.com/apache/arrow/pull/7535 >
