Wes McKinney created ARROW-1011:
-----------------------------------
Summary: [Format] Clarify in Layout.md what are the expectations
for buffer padding vs. in-memory data
Key: ARROW-1011
URL: https://issues.apache.org/jira/browse/ARROW-1011
Project: Apache Arrow
Issue Type: New Feature
Components: Format
Reporter: Wes McKinney
This has come up in https://github.com/apache/arrow/pull/673 and in prior
discussions.
The basic summary is that we should not write non-zero padding bytes in IPC
messages. However: one cannot in general rely on the padding being non-zero
when the data is in memory (for example: zero-copy slices of Arrow
arrays/vectors).
I think it would be good to clarify this point in Layout.md -- namely that what
gets written to the wire should be deterministic. However, in-memory algorithms
should not in general expect the padding region to have a particular value. As
an example, a popcount on a validity bitmap would want to exclude padding bytes
from the computation. Other elementwise SIMD operations are free to use the
padding bytes as they wish, with a known caveat.
cc [~cloud_fan]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
