[
https://issues.apache.org/jira/browse/PARQUET-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16585292#comment-16585292
]
Jonathan Underwood commented on PARQUET-1241:
---------------------------------------------
Here are some questions that we should probably answer.
Referring to my list of 3 implementations above:
* Should Arrow have implementations (1) and (2), or should (1) be replaced by
(2)? Replacing (1) with (2) will break backwards compatibility and wouldn't be
very customer friendly.
* Referring to the frame implementation (3), which of the following do we wish
to store in the compressed frame: (a) uncompressed column data size; (b)
uncompressed data checksum; (c) per-block uncompressed data checksums. In
Parquet, Items (a) and (b) are usually stored externally to the column data as
column metadata, I believe, so there would be redundancy. On the other hand,
storing (a) and (b) in the column data directly simplifies the use of the LZ4
library.
> Use LZ4 frame format
> --------------------
>
> Key: PARQUET-1241
> URL: https://issues.apache.org/jira/browse/PARQUET-1241
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-cpp, parquet-format
> Reporter: Lawrence Chan
> Priority: Major
>
> The parquet-format spec doesn't currently specify whether lz4-compressed data
> should be framed or not. We should choose one and make it explicit in the
> spec, as they are not inter-operable. After some discussions with others [1],
> we think it would be beneficial to use the framed format, which adds a small
> header in exchange for more self-contained decompression as well as a richer
> feature set (checksums, parallel decompression, etc).
> The current arrow implementation compresses using the lz4 block format, and
> this would need to be updated when we add the spec clarification.
> If backwards compatibility is a concern, I would suggest adding an additional
> LZ4_FRAMED compression type, but that may be more noise than anything.
> [1] https://github.com/dask/fastparquet/issues/314
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
