[
https://issues.apache.org/jira/browse/PARQUET-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16965631#comment-16965631
]
Martin Radev commented on PARQUET-1241:
---------------------------------------
Based on Yann's answer it looks like we can be safe in using the first four
bytes to determine whether the data is compressed using the lz4 frame
compression path. I think the key sentence from his answer is "The LZ4 frame
magic number can only be a valid sequence at the beginning of a new block if
said block follows some other blocks, or a dictionary, and is allowed to
reference into them.". There are no preceding blocks or dictionary.
Yann also adds "One generic way to handle such scenario is to interpret any
bytestream starting with the LZ4 frame magic header as a LZ4 frame, which is
almost always correct, and then backup to block mode if it fails and there
exists a slight chance that it could be a valid block.".
However, I think it would be better to rather not fallback to interpreting the
bytestream as a sequence of blocks. Frame decompression can also fail if the
checksum is not correct and having the fallback path would complicate code
somewhat.
Should I write the patches and push them for review?
> [C++] Use LZ4 frame format
> --------------------------
>
> Key: PARQUET-1241
> URL: https://issues.apache.org/jira/browse/PARQUET-1241
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-cpp, parquet-format
> Reporter: Lawrence Chan
> Priority: Major
>
> The parquet-format spec doesn't currently specify whether lz4-compressed data
> should be framed or not. We should choose one and make it explicit in the
> spec, as they are not inter-operable. After some discussions with others [1],
> we think it would be beneficial to use the framed format, which adds a small
> header in exchange for more self-contained decompression as well as a richer
> feature set (checksums, parallel decompression, etc).
> The current arrow implementation compresses using the lz4 block format, and
> this would need to be updated when we add the spec clarification.
> If backwards compatibility is a concern, I would suggest adding an additional
> LZ4_FRAMED compression type, but that may be more noise than anything.
> [1] https://github.com/dask/fastparquet/issues/314
--
This message was sent by Atlassian Jira
(v8.3.4#803005)