Hi Lasse,
Lasse Collin wrote:
I see it [the sync flush marker] is documented in lzlib to not be valid
outside lzlib-to-lzlib usage. I will change the xz man page.
Thank you.
On the other hand, I suppose the equivalent note in the liblzma
documentation should be preserved or updated to mention that it's not
fully lzlib compatible.
I had not considered the compatibility between liblzma and lzlib because of
the difference of formats, but I agree with you that it is useful to mention
in the documentation such incompatibility. On the positive side, I guess
that liblzma can decompress all .lz files (instead of "almost all").
I suggest to remove from container.h the text in the attached patch and
maybe replace it with a mention to the incompatibility between liblzma and
lzlib respect to the sync flush marker.
BTW, in line 866 of container.h there is this text about trailing data:
* The .lz format allows putting non-.lz data at the end of a file after at
* least one valid .lz member. That is, one can append custom data at the end
* of a .lz file and the decoder is required to ignore it.
The lzip format specification neither allows nor forbids trailing data.
Trailing data just happen. For example, a lzip data stream may be embedded
in another format. The lzip manual states that:
"Trailing data are in no way part of the lzip file format, but tools reading
lzip files are expected to behave as correctly and usefully as possible in
the presence of trailing data."
Therefore you may make liblzma treat trailing data as you see fit and still
meet the lzip format specification.
You may also wish to fix the comment at line 876 of container.h:
* one should ensure that none of the first four bytes of trailing data are
* equal to the magic bytes because lzip >= 1.20 requires it by default.
lzip >= 1.20 considers these by default:
4 matches with lzip_magic -> good header
2 or 3 matches with lzip_magic -> corrupt header
0 or 1 matches with lzip_magic -> trailing data
Version 1 format is definitely better, but I don't see why version 0
decoding support had to be removed. The format was produced by a few
stable lzip releases. Importantly, version 0 didn't have any horrible
defect like data corruption risk. It would have been better to avoid
the version 0 format completely, but mistakes happen to everyone, and
we have to live with them.
It certainly would have been better to avoid the version 0 format
completely, but it took me 3 months to develop lziprecover and complete the
design of the version 1 format. The version 0 "mistake" was fixed soon
enough that it seems to have affected nobody, and in fact almost nobody
seems to remember it. I don't see a reason to complicate lzip with support
for a version that nobody seems to be using.
As one more alternative to installing an old lzip version or sending a
(possibly private) file to you, they can simply use the latest xz.
I take your word for it. :-) If someone ever asks me how to decompress a
version 0 file, I'll advise him/her to use xz. But I would be surprised if
that happens because AFAIK nobody ever used version 0 for anything
important, not even myself. Even the ddrescue-1.9.tar.lz file you mention is
available in bzip2 format[1]. In the 9 years since I removed the last
decompression support for version 0 nobody has complained. Most of my
decompressors never supported version 0.
[1] http://download.savannah.gnu.org/releases/ddrescue/ddrescue-1.9.tar.bz2
Best regards,
Antonio.
--- /tmp/container.h~ 2025-04-18 20:51:06.000000000 +0200
+++ /tmp/container.h 2025-04-19 11:48:43.486070032 +0200
@@ -843,8 +843,7 @@
/**
* \brief Initialize .lz (lzip) decoder (a foreign file format)
*
- * This decoder supports the .lz format version 0 and the unextended .lz
- * format version 1:
+ * This decoder supports the .lz format versions 0 and 1:
*
* - Files in the format version 0 were produced by lzip 1.3 and older.
* Such files aren't common but may be found from file archives
@@ -853,12 +852,7 @@
* support for the format version 0 was removed in lzip 1.18.
*
* - lzip 1.3 added decompression support for .lz format version 1 files.
- * Compression support was added in lzip 1.4. In lzip 1.6 the .lz format
- * version 1 was extended to support the Sync Flush marker. This extension
- * is not supported by liblzma. lzma_code() will return LZMA_DATA_ERROR
- * at the location of the Sync Flush marker. In practice files with
- * the Sync Flush marker are very rare and thus liblzma can decompress
- * almost all .lz files.
+ * Compression support was added in lzip 1.4.
*
* Just like with lzma_stream_decoder() for .xz files, LZMA_CONCATENATED
* should be used when decompressing normal standalone .lz files.
