Hi Antonio, I filed an issue to pzstd and it looks like they will fix it..
https://github.com/facebook/zstd/issues/3255 Can you please close this zutils ticket? zutils is turning out to be essential as we switch to alternate compression methods. It is likely people will be working with different compression types regularly, and zutils has made the tools agnostic to the type of compression. Thank you, Chris -----Original Message----- From: Zutils-bug <zutils-bug-bounces+chris.jamboretz=intel....@nongnu.org> On Behalf Of Antonio Diaz Diaz Sent: Saturday, August 20, 2022 9:30 AM To: zutils-bug@nongnu.org Subject: Re: Skippable Frames in zstd files Hi Chris, Jamboretz, Chris wrote: > The zstd compression format allows for skippable frames > https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format. > md#skippable-frames > > pzstd uses this and puts the skippable frame at the beginning of the file. Thank you for reporting this. I think pzstd is making an improper use of skippable frames. The link above states that "Skippable frames defined in this specification are compatible with LZ4 ones". So it can't be known from the magic bytes if a file starting with a skippable frame is in zstd format or in lz4 format. In fact, https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#skippable-frames advises not to do what pzstd is doing: "For the purpose of facilitating identification, it is discouraged to start a flow of concatenated frames with a skippable frame. If there is a need to start such a flow with some user data encapsulated into a skippable frame, it's recommended to start with a zero-byte LZ4 frame followed by a skippable frame. This will make it easier for file type identifiers." > The function test_format in zutils.cc doesn't take this magic_number into > account and so reports the file as uncompressed. Detecting the 16 (!) magic numbers of skippable frames in test_format would be wrong because they do not really identify the zstd format. IMHO pzstd should follow the documentation of lz4 and start its files with a zero-byte zstd frame. Best regards, Antonio.