[
https://issues.apache.org/jira/browse/ARROW-300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392124#comment-16392124
]
Lawrence Chan edited comment on ARROW-300 at 3/9/18 2:09 AM:
-------------------------------------------------------------
What did we decide with this? Imho there's still a use case for compressed
arrow files due to the limited storage types in parquet. I don't really love
the idea of storing 8-bit or 16-bit ints in an INT32 and hand waving it away
with compression. I tried to hack it up with FixedLenByteArray but there are a
slew of complications with that, not to mention alignment concerns etc.
Anyways I'm happy to help on this, but I'm not familiar enough with the code
base to place it in the right spot. If we make a branch with some
TODOs/placeholders I can probably plug in more easily.
was (Author: llchan):
What did we decide with this? Imho there's still a use case for compressed
arrow files due to the limited storage types in parquet. I don't really love
the idea of storing 8-bit or 16-bit ints in an INT32 and hand waving it away
with compression.
Happy to help, but I'm not familiar enough with the code base to place it in
the right spot. If we make a branch with some TODOs/placeholders I can probably
plug in more easily.
> [Format] Add buffer compression option to IPC file format
> ---------------------------------------------------------
>
> Key: ARROW-300
> URL: https://issues.apache.org/jira/browse/ARROW-300
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Format
> Reporter: Wes McKinney
> Priority: Major
> Fix For: 0.10.0
>
>
> It may be useful if data is to be sent over the wire to compress the data
> buffers themselves as their being written in the file layout.
> I would propose that we keep this extremely simple with a global buffer
> compression setting in the file Footer. Probably only two compressors worth
> supporting out of the box would be zlib (higher compression ratios) and lz4
> (better performance).
> What does everyone think?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
