[
https://issues.apache.org/jira/browse/ARROW-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15416071#comment-15416071
]
Julien Le Dem commented on ARROW-253:
-------------------------------------
I'm familiar with the BitWeaving techniques and we should certainly look into
that in the future.
I'm not against bit packed vectors, I'm just wondering if they should be in V1
The other aspect we should consider is versioning of the spec and how we add
more vectors laters. We want to get to a point soon where all of the spec is
supported so we have to draw a line somewhere and limit the surface area for
implementors to ease adoption. Nothing prevents people to experiment with other
representations (as long as they stay internal). Those can be added to the spec
later.
some options:
- restrict int size in Arrow V1. Add bitpacked int vectors in V2.
- leave all widths (1 to 64) available in V1. Add converters to efficiently
unpack for engines that don't operate on packed vectors. (but someone needs to
create those).
> Int types should only have width of 8*2^n (8, 16, 32, 64)
> ---------------------------------------------------------
>
> Key: ARROW-253
> URL: https://issues.apache.org/jira/browse/ARROW-253
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Format
> Reporter: Julien Le Dem
>
> Since arrow is focused on CPU efficient operations and we want to keep the
> spec simple to the minimum required we should restrict the Int type to width
> of 1, 2, 4 or 8 bytes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
