[
https://issues.apache.org/jira/browse/PARQUET-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16965641#comment-16965641
]
Martin Radev commented on PARQUET-1241:
---------------------------------------
[~apitrou] can you point out other implementations but fastparquet and
parquet-mr?
I can have a look at how things are done there and open issues for the
corresponding projects.
Also, I suppose all implementations follow changes to the Parquet spec, so by
submitting patches and having a vote, many of the maintainers will be further
made aware.
I think the LZ4 frame format should be used as it leaves complications of the
encoding and necessary additional fields to the LZ4 implementations. At the
moment all parquet implementations have to make a choice what to use, how to
store additional necessary data like size and there has been some divergence.
> [C++] Use LZ4 frame format
> --------------------------
>
> Key: PARQUET-1241
> URL: https://issues.apache.org/jira/browse/PARQUET-1241
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-cpp, parquet-format
> Reporter: Lawrence Chan
> Priority: Major
>
> The parquet-format spec doesn't currently specify whether lz4-compressed data
> should be framed or not. We should choose one and make it explicit in the
> spec, as they are not inter-operable. After some discussions with others [1],
> we think it would be beneficial to use the framed format, which adds a small
> header in exchange for more self-contained decompression as well as a richer
> feature set (checksums, parallel decompression, etc).
> The current arrow implementation compresses using the lz4 block format, and
> this would need to be updated when we add the spec clarification.
> If backwards compatibility is a concern, I would suggest adding an additional
> LZ4_FRAMED compression type, but that may be more noise than anything.
> [1] https://github.com/dask/fastparquet/issues/314
--
This message was sent by Atlassian Jira
(v8.3.4#803005)