[gentoo-user] Speed up with pbzip2 or not!?
Hey guys! Yesterday I found this article http://gentoo-wiki.com/HOWTO_Speed_up_decompression_with_pbzip2 and was very happy, because I'm an owner of an Q6600. So I tried to reproduce the benchmark from the wiki article but got that bad results: test # time *bzip2* -d -kf linux-2.6.23.tar.bz2 real0m11.672s user0m11.306s sys 0m0.367s test # time *pbzip2* -d -p4 -kf linux-2.6.23.tar.bz2 real0m25.554s user0m24.862s sys 0m0.683s So the parallel version took more than the double time! To test whether this is a problem of my Pc I tested this on an Dual Core with the same result. An test with 7z was much better: test # time *7za* x -y linux-2.6.23.tar.bz2 real0m4.642s user0m8.379s sys 0m0.327s All tests where done in a tmpfs off 1GB on 2GB RAM. So my questions is what did I do wrong? I also tested it with different CFLAGS, but my CFLAGS are basically very conservative. (-O2 -march=prescott -pipe -fomit-frame-pointer)
Re: [gentoo-user] Speed up with pbzip2 or not!?
Am Sonntag, 27. Januar 2008 schrieb Justin: Yesterday I found this article http://gentoo-wiki.com/HOWTO_Speed_up_decompression_with_pbzip2 and was very happy, because I'm an owner of an Q6600. So I tried to reproduce the benchmark from the wiki article but got that bad results: So my questions is what did I do wrong? Your kernel is compiled with SMP support, isn't it? Bye... Dirk signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Speed up with pbzip2 or not!?
On Sun, 2008-01-27 at 10:58 +0100, Justin wrote: Hey guys! Yesterday I found this article http://gentoo-wiki.com/HOWTO_Speed_up_decompression_with_pbzip2 and was very happy, because I'm an owner of an Q6600. So I tried to reproduce the benchmark from the wiki article but got that bad results: test # time bzip2 -d -kf linux-2.6.23.tar.bz2 real0m11.672s user0m11.306s sys 0m0.367s test # time pbzip2 -d -p4 -kf linux-2.6.23.tar.bz2 real0m25.554s user0m24.862s sys 0m0.683s So the parallel version took more than the double time! To test whether this is a problem of my Pc I tested this on an Dual Core with the same result. An test with 7z was much better: test # time 7za x -y linux-2.6.23.tar.bz2 real0m4.642s user0m8.379s sys 0m0.327s So my questions is what did I do wrong? Did you compress it with pbzip2 in the first place? Pbzip2 can only speed up decompression of files created by pbzip2, as well. Read the docs, dude ... p7zip has got its own implementation of bzip2, which might be faster. signature.asc Description: This is a digitally signed message part
Re: [gentoo-user] Speed up with pbzip2 or not!?
Yes it is! But a question at this point: Where can I find the Enhanced Real Time Clock Support or is it gone in the 2.6.23? Dirk Heinrichs schrieb: Am Sonntag, 27. Januar 2008 schrieb Justin: Yesterday I found this article http://gentoo-wiki.com/HOWTO_Speed_up_decompression_with_pbzip2 and was very happy, because I'm an owner of an Q6600. So I tried to reproduce the benchmark from the wiki article but got that bad results: So my questions is what did I do wrong? Your kernel is compiled with SMP support, isn't it? Bye... Dirk -- gentoo-user@lists.gentoo.org mailing list
Re: [gentoo-user] Speed up with pbzip2 or not!?
Am Sonntag, 27. Januar 2008 schrieb Justin: But a question at this point: Where can I find the Enhanced Real Time Clock Support or is it gone in the 2.6.23? Run make menuconfig and use the search function (/). HTH... Dirk signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Speed up with pbzip2 or not!?
I did but without success. Thats why im asking!! Dirk Heinrichs schrieb: Am Sonntag, 27. Januar 2008 schrieb Justin: But a question at this point: Where can I find the Enhanced Real Time Clock Support or is it gone in the 2.6.23? Run make menuconfig and use the search function (/). HTH... Dirk -- gentoo-user@lists.gentoo.org mailing list
Re: [gentoo-user] Speed up with pbzip2 or not!?
Is there a way to use 7zip for decompression with emerge? Justin schrieb: Hey guys! Yesterday I found this article http://gentoo-wiki.com/HOWTO_Speed_up_decompression_with_pbzip2 and was very happy, because I'm an owner of an Q6600. So I tried to reproduce the benchmark from the wiki article but got that bad results: test # time *bzip2* -d -kf linux-2.6.23.tar.bz2 real0m11.672s user0m11.306s sys 0m0.367s test # time *pbzip2* -d -p4 -kf linux-2.6.23.tar.bz2 real0m25.554s user0m24.862s sys 0m0.683s So the parallel version took more than the double time! To test whether this is a problem of my Pc I tested this on an Dual Core with the same result. An test with 7z was much better: test # time *7za* x -y linux-2.6.23.tar.bz2 real0m4.642s user0m8.379s sys 0m0.327s All tests where done in a tmpfs off 1GB on 2GB RAM. So my questions is what did I do wrong? I also tested it with different CFLAGS, but my CFLAGS are basically very conservative. (-O2 -march=prescott -pipe -fomit-frame-pointer) -- gentoo-user@lists.gentoo.org mailing list
Re: [gentoo-user] Speed up with pbzip2 or not!?
Google is my friend: Device Drivers --- Character devices --- M Enhanced Real Time Clock Support Justin schrieb: I did but without success. Thats why im asking!! Dirk Heinrichs schrieb: Am Sonntag, 27. Januar 2008 schrieb Justin: But a question at this point: Where can I find the Enhanced Real Time Clock Support or is it gone in the 2.6.23? Run make menuconfig and use the search function (/). HTH... Dirk -- gentoo-user@lists.gentoo.org mailing list
Re: [gentoo-user] Speed up with pbzip2 or not!?
Justin wrote: Is there a way to use 7zip for decompression with emerge? Can this top-posting stop, please? Interleave, and snip. http://en.wikipedia.org/wiki/Top-posting Benno -- gentoo-user@lists.gentoo.org mailing list
Re: [gentoo-user] Speed up with pbzip2 or not!?
On Sun, 2008-01-27 at 11:40 +0100, Justin wrote: Is there a way to use 7zip for decompression with emerge? I fear, no easy one. Of course, since emerge is just a python script, you could rewrite it but 7z-syntax isn't compatible with bzip2 or gnu-tar so it will be some work. I also tried to use star as a drop-in replacement for gnu-tar, once. Didn't work well although their syntax is nearly identical, but just nearly... By the way: Please don't top-post (e.g. write your answers to a mail below the quote. It makes reading long threads with many quotes easier.) signature.asc Description: This is a digitally signed message part
Re: [gentoo-user] Speed up with pbzip2 or not!?
On Sonntag, 27. Januar 2008, Justin wrote: Thats a good point. Now it worked really fast. But then the questions is why should I use pbzip2 for decompression with portage? I think most tarballs are packed only with the normal compression algorithm! The WIKI articel pretends a gain of speed which wont be! and that is why you should never trust wiki-articles. Everybody can write them and say whatever they want. -- gentoo-user@lists.gentoo.org mailing list
Re: [gentoo-user] Speed up with pbzip2 or not!?
On Sunday 27 January 2008 18:29:56 Enrico Weigelt wrote: Why not an universal wrapper script ? maybe something like: magic-uncompress [-t format] [-c] [-o output] input This script could try to find out the input format automatically (the optional -t parameter allows to explicitly specifiy the format) Once we have this script, ebuilds could be rewritten step by step and no one (else than magic-uncompress) has to care about the actual commands behind. Some could be done w/ tar and zip, etc. You've just reinvented unpack(). It's been a part of portage for ages and is used in just about every ebuild in the tree... -- Bo Andresen signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Speed up with pbzip2 or not!?
* Florian Philipp [EMAIL PROTECTED] wrote: Hi folks, On Sun, 2008-01-27 at 11:40 +0100, Justin wrote: Is there a way to use 7zip for decompression with emerge? I fear, no easy one. Of course, since emerge is just a python script, you could rewrite it but 7z-syntax isn't compatible with bzip2 or gnu-tar so it will be some work. I also tried to use star as a drop-in replacement for gnu-tar, once. Didn't work well although their syntax is nearly identical, but just nearly... Why not an universal wrapper script ? maybe something like: magic-uncompress [-t format] [-c] [-o output] input This script could try to find out the input format automatically (the optional -t parameter allows to explicitly specifiy the format) Once we have this script, ebuilds could be rewritten step by step and no one (else than magic-uncompress) has to care about the actual commands behind. Some could be done w/ tar and zip, etc. cu -- - Enrico Weigelt== metux IT service - http://www.metux.de/ - Please visit the OpenSource QM Taskforce: http://wiki.metux.de/public/OpenSource_QM_Taskforce Patches / Fixes for a lot dozens of packages in dozens of versions: http://patches.metux.de/ - -- gentoo-user@lists.gentoo.org mailing list
Re: [gentoo-user] Speed up with pbzip2 or not!?
Enrico Weigelt schrieb:
* Florian Philipp [EMAIL PROTECTED] wrote:
Hi folks,
On Sun, 2008-01-27 at 11:40 +0100, Justin wrote:
Is there a way to use 7zip for decompression with emerge?
I fear, no easy one. Of course, since emerge is just a python script,
you could rewrite it but 7z-syntax isn't compatible with bzip2 or
gnu-tar so it will be some work. I also tried to use star as a drop-in
replacement for gnu-tar, once. Didn't work well although their syntax is
nearly identical, but just nearly...
Why not an universal wrapper script ?
maybe something like:
magic-uncompress [-t format] [-c] [-o output] input
This script could try to find out the input format automatically
(the optional -t parameter allows to explicitly specifiy the format)
Once we have this script, ebuilds could be rewritten step by step
and no one (else than magic-uncompress) has to care about the actual
commands behind.
Some could be done w/ tar and zip, etc.
cu
But than there is still the incompability problem with pbzip2 with bzip2
compressed files. The best solution at this point would be to compress
all official tarballs with pbzip2, because now and especially in future
parallel processing is the solution and second there is no problem
decompressing parallel compressed tarballs in single mode.
For an temporary work around I figured out this hack:
in /usr/lib/portage/bin/ebuild.sh there is a function defined for each
type of compression algorithm. I changed following lines
--- ebuild.sh.orig 2008-01-27 19:16:22.0 +0100
+++ ebuild.sh 2008-01-27 19:15:36.0 +0100
@@ -479,7 +479,7 @@
;;
bz2|bz)
if [ ${y} == tar ]; then
- 7za x -so ${srcdir}${x} | tar
xof - ${tar_opts}
+ bzip2 -dc ${srcdir}${x} | tar
xof - ${tar_opts}
assert $myfail
else
bzip2 -dc ${srcdir}${x}
${x%.*} || die $myfail
ebuild.sh.orig:
real0m16.962s
user0m16.289s
sys 0m2.780s
ebuild.sh:
real0m12.805s
user0m13.209s
sys 0m2.666s
With this trick you can change every kind of decompression command.
In this example you can save 25% of the time.
--
gentoo-user@lists.gentoo.org mailing list
Re: [gentoo-user] Speed up with pbzip2 or not!?
On Sun, 2008-01-27 at 13:54 +0100, Hemmann, Volker Armin wrote: On Sonntag, 27. Januar 2008, Justin wrote: Thats a good point. Now it worked really fast. But then the questions is why should I use pbzip2 for decompression with portage? I think most tarballs are packed only with the normal compression algorithm! The WIKI articel pretends a gain of speed which wont be! and that is why you should never trust wiki-articles. Everybody can write them and say whatever they want. Well, many eyes see much. It's all a matter of checking, just as with open source software... To the topic: I've unmerged pbzip2 after reading its docs. ... seemed too much trouble. I'll try it again when it is suitable as a drop-in replacement or when portage can make use of it. In the mean time, if I need good compression with more than one thread, I use p7zip's lzma implementation. By the way, as soon as I come into contact with some decent scripting languages (and no longer this closed source LabVIEW I currently have to work with), I'll try to build a wrapper around p7zip to create a drop-in replacement for gzip, bzip2 and zip. signature.asc Description: This is a digitally signed message part
