[
https://issues.apache.org/jira/browse/ARROW-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney reassigned ARROW-1011:
-----------------------------------
Assignee: Wenchen Fan
> [Format] Clarify requirements around buffer padding in validity bitmaps
> -----------------------------------------------------------------------
>
> Key: ARROW-1011
> URL: https://issues.apache.org/jira/browse/ARROW-1011
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Format
> Reporter: Wes McKinney
> Assignee: Wenchen Fan
> Fix For: 0.4.0
>
>
> This has come up in https://github.com/apache/arrow/pull/673 and in prior
> discussions.
> The basic summary is that we should not write non-zero padding bytes in IPC
> messages. However: one cannot in general rely on the padding being non-zero
> when the data is in memory (for example: zero-copy slices of Arrow
> arrays/vectors).
> I think it would be good to clarify this point in Layout.md -- namely that
> what gets written to the wire should be deterministic. However, in-memory
> algorithms should not in general expect the padding region to have a
> particular value. As an example, a popcount on a validity bitmap would want
> to exclude padding bytes from the computation. Other elementwise SIMD
> operations are free to use the padding bytes as they wish, with a known
> caveat.
> cc [~cloud_fan]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
