[
https://issues.apache.org/jira/browse/ARROW-300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667983#comment-15667983
]
Uwe L. Korn commented on ARROW-300:
-----------------------------------
Given my latest (sadly internal) performance tests, I'm not so sure about the
benefit of a compressed arrow file format. For me the main distinction is that
Parquet provides efficient storage (with the tradeoff of not being to randomly
access a single row) and Arrow random access, both for columnar data.
The one point where I see an Arrow file format as beneficial is where you need
random access to its data but cannot load it fully into RAM but instead use a
memory mapped file. If you add compression (either column-wise or whole-file
level), you cannot memorymap it anymore.
The only point where I can see that having columnar compression for Arrow
batches is better than on the whole file layer is that it actually produces
better compression behaviour. This means that doing compression on a per-column
basis can be parallelised independently of the underyling algorithm thus
leading to better CPU usage. Furthermore the compression may be better if done
on a column level (with a sufficient number of rows) as the data inside a
column is very similar thus leading to smaller compression dictionaries and
better compresssion ratios at the end. Both things mentioned are just
assumptions that should be tested before being implemented.
> [Format] Add buffer compression option to IPC file format
> ---------------------------------------------------------
>
> Key: ARROW-300
> URL: https://issues.apache.org/jira/browse/ARROW-300
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Format
> Reporter: Wes McKinney
>
> It may be useful if data is to be sent over the wire to compress the data
> buffers themselves as their being written in the file layout.
> I would propose that we keep this extremely simple with a global buffer
> compression setting in the file Footer. Probably only two compressors worth
> supporting out of the box would be zlib (higher compression ratios) and lz4
> (better performance).
> What does everyone think?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
