[ 
https://issues.apache.org/jira/browse/ARROW-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009561#comment-16009561
 ] 

Wes McKinney commented on ARROW-1011:
-------------------------------------

It's possible we could draw a distinction between data in memory and data that 
is put on the wire. 

As an example, consider an int32 array with 10 elements and no nulls. Then we 
may allocate a single buffer:

- data buffer: 64 bytes

in the case of the data buffer, only 40 bytes have a specified value. It might 
be that an implementation zeros out all of its memory. But when we write this 
data to the streaming format, it would be better for the padding bytes to be 
zeros. So when we are unloading this array for writing on the wire:

* data buffer: 64 bytes length, 40 initialized bytes
* write first 40 bytes
* write 24 bytes of 0s

this replaces the current logic where we write the contents of the buffer 
exactly, independent of the "effective range".

As one practical matter, it's not practical to write 0 bytes for the contents 
of null cells, per our prior discussion on GitHub. 

I don't this is a big deal; we can always stick with the current language in 
the document and say that the padding bytes may have any value.

> [Format] Clarify in Layout.md what are the expectations for buffer padding in 
> IPC vs. in-memory data
> ----------------------------------------------------------------------------------------------------
>
>                 Key: ARROW-1011
>                 URL: https://issues.apache.org/jira/browse/ARROW-1011
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: Format
>            Reporter: Wes McKinney
>
> This has come up in https://github.com/apache/arrow/pull/673 and in prior 
> discussions. 
> The basic summary is that we should not write non-zero padding bytes in IPC 
> messages. However: one cannot in general rely on the padding being non-zero 
> when the data is in memory (for example: zero-copy slices of Arrow 
> arrays/vectors).
> I think it would be good to clarify this point in Layout.md -- namely that 
> what gets written to the wire should be deterministic. However, in-memory 
> algorithms should not in general expect the padding region to have a 
> particular value. As an example, a popcount on a validity bitmap would want 
> to exclude padding bytes from the computation. Other elementwise SIMD 
> operations are free to use the padding bytes as they wish, with a known 
> caveat. 
> cc [~cloud_fan]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Reply via email to