Re: [Zutils-bug] [RFE] Please consider adding support for lbzip2
>> 3) Necessity for additional configuration. Zutils works fine after >>replacing bzip2 with lbzip2 (substituting "bzip2" with "lbzip2" in >>decompressor_names[] in zutils.h). There is no need for any >>additional configuration. > > > This is easy to do, but too rigid. For example, in my single-processor > machine I have no parallel compressors installed (except plzip, but only > because I develop it). That was meant only as a proof that lbzip2 doesn't need any more configuration than bzip2. I realise that a real solution would be more sophisticated. >> I can prepare a patch if needed. > > > Thanks for the offering, but I have already some work done borrowing from > GNU moe and the moerc file. Now that I feel the pressure I'll try to get it > finished ASAP. :-) Thank you. -- Mikolaj ___ Zutils-bug mailing list Zutils-bug@nongnu.org https://lists.nongnu.org/mailman/listinfo/zutils-bug
Re: [Zutils-bug] [RFE] Please consider adding support for lbzip2
Hello Mikolaj. Mikolaj Izdebski wrote: Please consider adding support for lbzip2 in Zutils. Sure. I have already planned (from long ago) to add support for parallel compressors to zutils. It is simply that I haven't yet found the time to implement it. 3) Necessity for additional configuration. Zutils works fine after replacing bzip2 with lbzip2 (substituting "bzip2" with "lbzip2" in decompressor_names[] in zutils.h). There is no need for any additional configuration. This is easy to do, but too rigid. For example, in my single-processor machine I have no parallel compressors installed (except plzip, but only because I develop it). I plan a flexible way of choosing decompressors, with a configuration file setting the defaults and command line options for overriding them. I can prepare a patch if needed. Thanks for the offering, but I have already some work done borrowing from GNU moe and the moerc file. Now that I feel the pressure I'll try to get it finished ASAP. :-) Best regards, Antonio. ___ Zutils-bug mailing list Zutils-bug@nongnu.org https://lists.nongnu.org/mailman/listinfo/zutils-bug
[Zutils-bug] [RFE] Please consider adding support for lbzip2
Please consider adding support for lbzip2 in Zutils. lbzip2 [1] is a free software implementation of bzip2 which aims for reliability and performance. It aims to be fully compatible with bzip2, at both file format and command-line interface levels. It's closer to GNU philosophy than bzip2 (bzip2 is under permissive license and uses term "open-source", while lbzip2 is under GPLv3+ and describes itself as free software). It is also closer technically to GNU system (it uses GNU build system and Gnulib). Adding support for alternative bzip2 decompressors was already suggested [2]. I believe that using lbzip2 instead of bzip2 would be advantageous. In [2] you mentioned three disadvantages of using independent bzip2 decompressors, I'll try to deny them all. 1) Portability. lbzip2 is written in C89 and it uses GNU build system and Gnulib. It works on virtually any hardware architecture and modern-enough system. It even does some crazy portability things like limiting length of string literals to 509 characters. I consider lbzip2 to be more portable than Zutils. 2) Resource consumption. Even when using single thread lbzip2 uses much less time (CPU power) to decompress bz2 files than bzip2 does. Memory usage is almost as low as bzip2. An example benchmark is shown below. lbzip2 performance scales almost linearly with number of processors available in the system, up to more than one hundred [3] -- there is only little overhead of using multiple threads. 3) Necessity for additional configuration. Zutils works fine after replacing bzip2 with lbzip2 (substituting "bzip2" with "lbzip2" in decompressor_names[] in zutils.h). There is no need for any additional configuration. If you absolutely want to make sure that resource consumption remains minimal you can force usage of a single thread by lbzip2, although I wouldn't recommend doing that. Users can set their individual preferences in LBZIP2 environmental variable. Benchmark showing performance of lbzip2 on single-processor machine compared to bzip2 (decompressing latest Linux source tarball): Command being timed: "lbzcat linux-3.9.7.tar.bz2" User time (seconds): 13.08 System time (seconds): 0.98 Percent of CPU this job got: 98% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:14.29 Maximum resident set size (kbytes): 6200 Command being timed: "bzcat linux-3.9.7.tar.bz2" User time (seconds): 17.12 System time (seconds): 0.18 Percent of CPU this job got: 98% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:17.59 Maximum resident set size (kbytes): 4060 This is not a scientific benchmark, but you are free to conduct your own. I'm sure the results will be similar. I can prepare a patch if needed. -- Mikolaj Izdebski [1] http://github.com/kjn/lbzip2 [2] http://lists.nongnu.org/archive/html/zutils-bug/2011-02/msg1.html [3] http://lacos.hu/lbzip2-scaling/scaling.html ___ Zutils-bug mailing list Zutils-bug@nongnu.org https://lists.nongnu.org/mailman/listinfo/zutils-bug
