Re: [Lzip-bug] On Windows unpacking does not use all cores
Romano wrote: Yesterday I compiled latest lzlib 1.9 and plzip 1.7 under cygwin(also latest, just installed yesterday) on Windows 7 and as a 64bit both. It compiled without any errors or warning and tool also work fine. During packing it is able to utilize all CPU cores to 100% so multi threading works. Same with testing using -t flag. However, when I actually try to unpack with -d it never even peak above 50%. This despite -n option, even if I double the number to -n 8. On parallel, until now I always used FreeArc's internal 4x4:lzma which always fully utilized my cpu and it shows as during unpacking without io limitation it could reach ~200Mib/s. I don't use Windows, so I can't test your executable. (BTW, please, don't spam the list with huge unsolicited files). The fact that plzip can use all cores while testing makes me suspect of some I/O problem/idiosyncrasy. See for example this thread on the MinGW list: sourceforge.net sourceforge.netI am aware of blocks concept as well, tool also did not utilized all CPU with smaller -B block and big enough file, and I know for sure its not my HDD limiting it because first it is quicker than output and FreeArc can still utilize its max, but also because it does not utilize full CPU even if using stdin/stdout. Even if decompressing from a regular file to /dev/null ? Yes - it seems /dev/null is treated as you said as non seekable - I noticed the same on linux decompressing to null device, you can check yourself. I was actually to report the very same thing and made some test with some big amount of data only to find out it's intended behaviour. But not all in vein - it turned out that decompressing to standard file removes limitations. ___ Lzip-bug mailing list Lzip-bug@nongnu.org https://lists.nongnu.org/mailman/listinfo/lzip-bug
Re: [Lzip-bug] On Windows unpacking does not use all cores
Romano wrote: May I ask one last thing, is there a reason to use LC - literal context parameter lower than 8? Default is 3 and in lzma2 its not even allowed above 4. But lc8 give better ratio(~2.2% in my tests and many others on fileforums) for most data especially game repacks. I changed parameter in the header and it seems workign fine(I guess your lzlib is based on lzma1?), but I never understood why would 3 be the default promoted choice especially since lc does not affect speed and seems to be only benefit at 8. The reason to not use lc = 8 is that it requires 768 KiB of fast RAM vs the 24 KiB required by lc = 3. This would make decompression slow on machines with, say, 64 KiB of cache. It also does not increase the compression ratio of tarballs by much. I just made a little test with a bunch of tarballs: Total size of tar files = 2_649_872_384 Compressed sizes: level -0 -6 -9 lc = 3576_393_065391_449_171367_964_614 lc = 8569_160_271388_354_250365_461_202 -1.2% -0.8% -0.7 Lastly, pls consider to add more lzma options to the command line in next versions. Specifically mc and lc parameter would be very useful. Adding backward-incompatible options is out of the scope of lzip. As you can read here[1], "The lzip format is designed for long-term archiving. Therefore it excludes any unneeded features that may interfere with the future extraction of the decompressed data." [1] http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance There are lots of tweaks that can increase slightly the compression ratio of some kinds of data, generally at the cost of increasing resource consumption (memory/cpu), and worse, at the cost of producing incompatible files. There is also one tweak that 4x4 use that have significant effect on speed of compression: it test incoming data that are to be processed by lzma with some fast "order 0 model" first. If it cannot compress better than to 99% then data are copied instead. This give anywhere between 50%-300% speed increase. These clever tweaks are fine for experimenting, but are also out of the scope of a stable standard general-purpose compressor like lzip. See for example how neither gzip nor bzip2 have made such changes to their formats. Best regards, Antonio. ___ Lzip-bug mailing list Lzip-bug@nongnu.org https://lists.nongnu.org/mailman/listinfo/lzip-bug
Re: [Lzip-bug] On Windows unpacking does not use all cores
Hello Romano, Romano wrote: I found that anything above -B 32Mi start decreasing number of cores utilized upon decompression(on 4 core cpu). Up to -B 32Mi -s 32Mi (-m 32) will still load 100% CPU on -d -c command to stdout, but already -B 48Mi get it down to 80% and 64Mi+ go back to 25-50%. This is by testing plzip on command line alone not in conjuction with FreeArc. This is caused by the 32 MiB buffering limit of plzip intended to prevent it from using too much RAM when decompressing large blocks to a non-seekable destination. See http://www.nongnu.org/lzip/manual/plzip_manual.html#Memory-requirements "For decompression of a regular file to a non-seekable file or to standard output; the dictionary size plus up to 32 MiB." The effect is more notable the more cores one uses. For example I need to use '-B 128MiB' to reduce cpu use to 133% (66% as reported by Windows) on my dual core linux system. Decompressing to a regular file should give you full decompression speed: http://www.nongnu.org/lzip/manual/plzip_manual.html#Program-design "When decompressing from a regular file, the splitter is removed and the workers read directly from the input file. If the output file is also a regular file, the muxer is also removed and the workers write directly to the output file. With these optimizations, the use of RAM is greatly reduced and the decompression speed of large files with many members is only limited by the number of processors available and by I/O speed." Best regards, Antonio. ___ Lzip-bug mailing list Lzip-bug@nongnu.org https://lists.nongnu.org/mailman/listinfo/lzip-bug
Re: [Lzip-bug] On Windows unpacking does not use all cores
Hello Romano, Romano wrote: Yesterday I compiled latest lzlib 1.9 and plzip 1.7 under cygwin(also latest, just installed yesterday) on Windows 7 and as a 64bit both. It compiled without any errors or warning and tool also work fine. During packing it is able to utilize all CPU cores to 100% so multithreading works. Same with testing using -t flag. Howerver, when I actually try to unpack with -d it never even peak above 50%. This despite -n option, even if I double the number to -n 8. On parallel, until now I always used FreeArc's internal 4x4:lzma which always fully utilized my cpu and it shows as during unpacking without io limitation it could reach ~200Mib/s. I don't use Windows, so I can't test your executable. (BTW, please, don't spam the list with huge unsolicited files). The fact that plzip can use all cores while testing makes me suspect of some I/O problem/idiosyncrasy. See for example this thread on the MinGW list: http://sourceforge.net/p/mingw/mailman/message/35749043/ I am aware of blocks concept as well, tool also did not utilized all CPU with smaller -B block and big enough file, and I know for sure its not my HDD limiting it because first it is quicker than output and FreeArc can still utilize its max, but also because it does not utilize full CPU even if using stdin/stdout. Even if decompressing from a regular file to /dev/null ? I hope this help you and thanks for a great tool you did. You are welcome. (PS: Also, I dont know if this is a bug in plzip, but if I compress under Freearc and click "cancel", first 3 threads eventually cancel as they should but last one remain with CPU at 25% forever - thus plzip never quit.) Sorry, but I am not familiar with FreeArc. Do you know if it is plzip or FreeArc the one stuck in the infinite loop? Plzip should exit as soon as its standard input is closed by FreeArc. Best regards, Antonio. ___ Lzip-bug mailing list Lzip-bug@nongnu.org https://lists.nongnu.org/mailman/listinfo/lzip-bug
