Hello Erik, Erik Jan Tromp wrote:
I believe I've found a bug in plzip. Apologies for rambling.
Thanks for reporting the issue. Better safe than sorry. :-)
software: Slackware 14.2 (32 bit) gcc (GCC) 5.3.0 plzip 1.6 hardware: Intel(R) Core(TM) i7 CPU 930 @ 2.80GHz quad core with hyperthreading 6GB RAM (not a typo)
According to [1], compressing at level 9 requires about 32 * 11 + 64 * 3 = 544 MiB. For 8 threads this takes 4352 MiB (4.25 GiB). I suspect the problem is that a process can only use less than 4 GiB on a 32 bit system.
[1] http://www.nongnu.org/lzip/manual/plzip_manual.html#Memory-requirements
test: Compress each tarball with 'plzip -vf9 foo.tar' Tarballs 421969920 bytes& larger: fail to compress Tarballs 305926656 bytes& smaller: compress successfully
This is because compressing a file of 306 MB at level 9 can use only 5 threads. See [2].
[2] http://www.nongnu.org/lzip/manual/plzip_manual.html#Minimum-file-sizes
Tests were also performed on the following hardware using 'plzip -vf9 foo.tar' with no errors.
All of them seem to either be 64 bit or have too few processors to reach the per process memory limit of 32 bit. Am I wrong?
Please note that while specifying '-n'& '-s' options are functional solutions, they are unacceptable for the intended use case. Regardless of a machine's hardware, the resulting compressed files must be identical/predictable.
If you want identical results on 32 and 64 bit machines, I guess you'll need to limit the number of threads. Limiting the dictionary size may not be enough on 32 bit systems with many cores.
Best regards, Antonio. _______________________________________________ Lzip-bug mailing list [email protected] https://lists.nongnu.org/mailman/listinfo/lzip-bug
