Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2013-01-03 Thread Samuli Suominen

On 03/01/13 03:28, Rick Zero_Chaos Farina wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 01/02/2013 06:11 PM, Pacho Ramos wrote:

El lun, 29-10-2012 a las 10:39 -0400, Rich Freeman escribió:

On Mon, Oct 29, 2012 at 10:13 AM, Rick Zero_Chaos Farina
zeroch...@gentoo.org wrote:

I'm confident no one would
attempt to block my adding eselect-bzip2 to the tree (aside from my poor
coding skills),


++


but would anyone be interested in blocking using lbzip2
by default?  It seems pretty safe and I've done dozens of full system
builds etc.


Why not just make it an option to start, advertise it by all means,
and then see how it turns out with volunteers before we make it the
default.  Going from nobody-has-heard-of-it to default overnight seems
a bit much.

Rich




How this ended finally? :/

It ended with my setting PORTAGE_BZIP2_COMMAND=lbzip2 in my profile
and moving on with my life.  I am very not good at eselect scripts and
I've not had the time to dig at it. You are welcome to work on the
eselect script if you like.


Feel free to steal the code used in eselect-pinentry.

It was originally written by mgorny for eselect-sh, but then later 
modified by me, and has not had a single bug ever since.





Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2013-01-03 Thread Diego Elio Pettenò
On 03/01/2013 11:30, Samuli Suominen wrote:

 
 Feel free to steal the code used in eselect-pinentry.
 
 It was originally written by mgorny for eselect-sh, but then later
 modified by me, and has not had a single bug ever since.

Can I reiterate that it'd be cool to have an eselect tools? :)

-- 
Diego Elio Pettenò — Flameeyes
flamee...@flameeyes.eu — http://blog.flameeyes.eu/



Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2013-01-02 Thread Pacho Ramos
El lun, 29-10-2012 a las 10:39 -0400, Rich Freeman escribió:
 On Mon, Oct 29, 2012 at 10:13 AM, Rick Zero_Chaos Farina
 zeroch...@gentoo.org wrote:
  I'm confident no one would
  attempt to block my adding eselect-bzip2 to the tree (aside from my poor
  coding skills),
 
 ++
 
  but would anyone be interested in blocking using lbzip2
  by default?  It seems pretty safe and I've done dozens of full system
  builds etc.
 
 Why not just make it an option to start, advertise it by all means,
 and then see how it turns out with volunteers before we make it the
 default.  Going from nobody-has-heard-of-it to default overnight seems
 a bit much.
 
 Rich
 
 

How this ended finally? :/

Thanks for the info!


signature.asc
Description: This is a digitally signed message part


Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2013-01-02 Thread Rick Zero_Chaos Farina
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 01/02/2013 06:11 PM, Pacho Ramos wrote:
 El lun, 29-10-2012 a las 10:39 -0400, Rich Freeman escribió:
 On Mon, Oct 29, 2012 at 10:13 AM, Rick Zero_Chaos Farina
 zeroch...@gentoo.org wrote:
 I'm confident no one would
 attempt to block my adding eselect-bzip2 to the tree (aside from my poor
 coding skills),

 ++

 but would anyone be interested in blocking using lbzip2
 by default?  It seems pretty safe and I've done dozens of full system
 builds etc.

 Why not just make it an option to start, advertise it by all means,
 and then see how it turns out with volunteers before we make it the
 default.  Going from nobody-has-heard-of-it to default overnight seems
 a bit much.

 Rich


 
 How this ended finally? :/
It ended with my setting PORTAGE_BZIP2_COMMAND=lbzip2 in my profile
and moving on with my life.  I am very not good at eselect scripts and
I've not had the time to dig at it. You are welcome to work on the
eselect script if you like.

 Thanks for the info!
 

- -Zero
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iQIcBAEBAgAGBQJQ5N63AAoJEKXdFCfdEflKdn0P/R9W6idHgpd+n7y5nyKQM9si
noHmFC6KHJSZJnjO8BlrhsN2HZrgZyPqPIlWLn5VwK+KcDTh0s1y4UGGh22a93qD
JodVbmmSJlnWA+EkzNloG+nTk1FxJYnmoL7o6/dhqv5x1Q/FD/xf0rAGCgTvDPSb
RuoMbCfkgSwZZlSSJtHr7Xd3w1psAgVemXNcG1C49AmnNrrmEkvawk5SNzg5Ruyn
UVMKSbLt1zEkRmUZHWS+lizN+GjFMVlaPEb6KK7ELbYWEMIjBihZ0WqdSWQ5Yxcn
yaZN9RTM2xel9tS+PEZjxwCBwb4ndXxuj51zK91WeTeWL+1wvwRXQ0nL+jFi3WfA
9nIHe26Cop5btYnCw2/5wlmKYReAxasql0ZVS8mrob5IgiptMIrxCAhCq8raJVZn
VD3+GiWzHYQRBUEkwQq/Qq7n7d5hgyd8l2/CCSgNUxmNbtFLoNWxPK5aVvEeb5EH
zyLH0eOQp4JA2ViZtnYgkviFRaEr9pa+Kc2S4BbvPTqYOX7Q7Ah2LMOw8uXQMefB
cNlVBa6ar5feAybWkmKH+c9n6mjAU9iCWyuoATSUMpSQ1dPmDnmKmxst4kARgMVK
Ou1F4mREQk7N23AqAnREF3b4RNLQqv6ff5GuxG4COjXZpIynFcm4o/x4nnzan6xr
ALCXZ5LIRk5D5K//oYJU
=yYs4
-END PGP SIGNATURE-



Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-10-31 Thread Sergey Popov
29.10.2012 18:39, Rich Freeman wrote:
 On Mon, Oct 29, 2012 at 10:13 AM, Rick Zero_Chaos Farina
 but would anyone be interested in blocking using lbzip2
 by default?  It seems pretty safe and I've done dozens of full system
 builds etc.
 
 Why not just make it an option to start, advertise it by all means,
 and then see how it turns out with volunteers before we make it the
 default.  Going from nobody-has-heard-of-it to default overnight seems
 a bit much.
 

+1 for that

-- 
Best regards, Sergey Popov
Gentoo Linux Developer
Desktop-effects project lead



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-10-29 Thread Rick Zero_Chaos Farina
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 09/27/2012 05:23 AM, Piotr Szymaniak wrote:
 On Wed, Sep 26, 2012 at 01:43:27PM -0700, Matt Turner wrote:
 On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol mike...@gmail.com wrote:
 A few months ago, I filed bug 423651 to ask that bzip2 on the install
 media be replaced with
  pbzip2. It was closed a short while later, telling me that it'd
 involve changing what's kept in @system, and that had to be discussed
 here, rather than in a bug report.

 If we're going to ship a parallel bzip2 implementation, it should be
 lbzip2 and not pbzip2.

 lbzip2 can decompress bz2 archives with multiple threads that haven't
 been compressed with lbzip2/pbzip2.
 
 Afair I'm using PORTAGE_BZIP2_COMMAND with lbzip2 and it works fine.
 Also some time ago I've changed a bit the (famous?) stage4 backup
 script from g-wiki to support parallel gz/bz2 implementations (simple
 check if there pbzip2/lbzip2/foobar installed and if yes, use it instead
 of normal gzip/bzip2).

I've been testing PORTAGE_BZIP2_COMMAND=lbzip2 for the few weeks since
this thread as well, and I have to say it's been great.
 
 Maybe portage should be like my stage4 mod? If it finds some parallel
 (de)compressor it should use it, if not fallback to standard gzip/bzip2?

The releng team has begun using lbzip2 for decompressing things as much
as possible for speeding up catalyst builds, and speaking only for
myself I think it's been useful.  I spoke to zmedico about replacing the
default portage decompresser and he suggested far more use could be
attained by simply creating an eselect-bzip2 (possibly defaulting to
lbzip2 as this thread seemed to agree).  I'm confident no one would
attempt to block my adding eselect-bzip2 to the tree (aside from my poor
coding skills), but would anyone be interested in blocking using lbzip2
by default?  It seems pretty safe and I've done dozens of full system
builds etc.

Thanks,
Zero_Chaos
 
 
 Piotr Szymaniak.
 

-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQIcBAEBAgAGBQJQjo79AAoJEKXdFCfdEflKQNQQAIur3Mx4eD6kVv3US6kDkKpn
Z8aHyZqWIOws9VqcPCJUb1c1bn8aj2QTjL63obh//rJby1Qd1P3X3OA9hKfaiyiJ
hbgWRu5RkMSNPIdXhaslD3h/kL78BPr9eVz9O3R/TMp01puz4su0j1JVOtAT5Ny2
qAVof+H6RC5aCP2pKjIkvDq9vma/jd9QT98rM3UIw3BorBpkl0vTTa+sLtGA+U6p
mBkYZJFUUnkW3AtNC1ucPvQS9a7NwOxH5EDGEy4QeSJh9nvQZ8zCFivNtr4SUS6A
SGV0j/PB0Uqf42y1NOBfFSEQoQGT3aIlZzkbGdJKf6/jIdGPHhxWt3npQ6U6gOg9
TA/8zCJkJk0GZKAIq+K/y8STczeuIJ2hgrHUup4/I6CYwI66eBRiqnayNEmah7Cy
rpqR7xwqh+F2JrjljNYfE9qVQKwqhLU5MsOONlqRS+FPw+SuxxbJM6BiLCk4s0Qz
GPs5PXTz92MzzQ18XUuMq53bi9IaEmJJa8E83IBm3PD4zq67KsiWd2YxQo/JYX4p
ZTBayLabbY6PRwMVx0k9jKG7aOOKFZ5t2k5BmZDv6JBD2Sww9q6+xUPoTDwsZPgZ
pVv2m3AwaOXbGpvMrpqmzOTxbovUALp9WhdyTxlR5NmbTeXIuaKW/lQTU0cl/LKP
ACg4IADjv3hQz7gfsHAP
=hCz9
-END PGP SIGNATURE-



Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-10-29 Thread Rich Freeman
On Mon, Oct 29, 2012 at 10:13 AM, Rick Zero_Chaos Farina
zeroch...@gentoo.org wrote:
 I'm confident no one would
 attempt to block my adding eselect-bzip2 to the tree (aside from my poor
 coding skills),

++

 but would anyone be interested in blocking using lbzip2
 by default?  It seems pretty safe and I've done dozens of full system
 builds etc.

Why not just make it an option to start, advertise it by all means,
and then see how it turns out with volunteers before we make it the
default.  Going from nobody-has-heard-of-it to default overnight seems
a bit much.

Rich



Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-27 Thread Florian Philipp
Am 26.09.2012 23:53, schrieb Michael Mol:
 On Wed, Sep 26, 2012 at 5:27 PM, Florian Philipp li...@binarywings.net 
 wrote:
 Am 26.09.2012 22:43, schrieb Matt Turner:
 On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol mike...@gmail.com wrote:
 A few months ago, I filed bug 423651 to ask that bzip2 on the install
 media be replaced with
  pbzip2. It was closed a short while later, telling me that it'd
 involve changing what's kept in @system, and that had to be discussed
 here, rather than in a bug report.

 If we're going to ship a parallel bzip2 implementation, it should be
 lbzip2 and not pbzip2.

 lbzip2 can decompress bz2 archives with multiple threads that haven't
 been compressed with lbzip2/pbzip2.


 This seems relevant, especially comment 12ff:
 https://bugs.gentoo.org/show_bug.cgi?id=309683

 For further anecdotal evidence: I've used pbzip2 with USE=symlink for
 several months now and never had trouble with it. Checking out lbzip2
 now. I noticed it doesn't install a bunzip2 symlink.
 
 Piotr Szymaniak asked me about lbzip2, and I bounced the question over
 to my friend. He didn't investigate it deeply; it crashed (OOM or
 something else, I don't know) when he tried it on a large file. Could
 have been from 2GB to 2TB, from what he has laying around. I don't
 know; I didn't get that one in writing. :)
 
 But if it proves to be stable for small and very large files, I'd have
 no complaint. :)
 

I just encountered this:

bzip2 -c /srv/qemu/hpwin.img /dev/null
bzip2:
/var/tmp/portage/app-arch/lbzip2-2.2/work/lbzip2-2.2/src/encode.c:794:
generate_initial_trees: Assertion `a  b' failed.

Something in that file is upsetting lbzip2. I'm investigating.



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-27 Thread Tobias Klausmann
Hi! 

On Wed, 26 Sep 2012, Chí-Thanh Christopher Nguyễn wrote:
 A different question is whether in the cases where parallel bzip2 makes
 sense, is it really the best solution? xz is outperforming bzip2's
 compression ratio for large files (for an informal comparison, see bug
 434350). And xz is faster at decompression, which offsets the parallel
 advantage to some degree.

As for the performance side of things: 
http://blog.i-no.de/archives/2008/05/08/index.html#e2008-05-08T16_35_13.txt

Yes, that's four years old and needs to be redone with modern
implementations (and machines). Will do so this weekend (I hope).

Regards,
Tobias

-- 
printk (KERN_ALERT You are screwed!  ...);
linux-2.6.6/arch/i386/kernel/efi.c



Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-27 Thread Piotr Szymaniak
On Wed, Sep 26, 2012 at 01:43:27PM -0700, Matt Turner wrote:
 On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol mike...@gmail.com wrote:
  A few months ago, I filed bug 423651 to ask that bzip2 on the install
  media be replaced with
   pbzip2. It was closed a short while later, telling me that it'd
  involve changing what's kept in @system, and that had to be discussed
  here, rather than in a bug report.
 
 If we're going to ship a parallel bzip2 implementation, it should be
 lbzip2 and not pbzip2.
 
 lbzip2 can decompress bz2 archives with multiple threads that haven't
 been compressed with lbzip2/pbzip2.

Afair I'm using PORTAGE_BZIP2_COMMAND with lbzip2 and it works fine.
Also some time ago I've changed a bit the (famous?) stage4 backup
script from g-wiki to support parallel gz/bz2 implementations (simple
check if there pbzip2/lbzip2/foobar installed and if yes, use it instead
of normal gzip/bzip2).

Maybe portage should be like my stage4 mod? If it finds some parallel
(de)compressor it should use it, if not fallback to standard gzip/bzip2?


Piotr Szymaniak.
-- 
 - Jeden hamburger na dziesiec ci zaszkodzi. Jeden moj stary przyjaciel
to sprawdzil.  Zjadal dziewiec hamburgerow i byl idealnie zdrowy, a gdy
probowal zjesc dziesiatego, z miejsca dostawal torsji.
  -- Graham Masterton, Night Warriors


signature.asc
Description: Digital signature


Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-27 Thread Ulrich Mueller
 On Wed, 26 Sep 2012, Michael Mol wrote:

 A few months ago, I filed bug 423651 to ask that bzip2 on the
 install media be replaced with pbzip2. It was closed a short while
 later, telling me that it'd involve changing what's kept in @system,
 and that had to be discussed here, rather than in a bug report.

We need to be careful when we replace such standard tools. Often the
replacement isn't completely compatible. For example, pbzip2 suffers
from the same bug as pigz [1] when it encounters a zero-padded tarball:

   $ echo foo | bzip2 | dd conv=sync 2/dev/null | pbzip2 -d
   foo
   pbzip2: *ERROR during BZ2_bzDecompress - trailing garbage: ret=4; block=0; 
seq=0; isLastInSeq=1; avail_in=472
   Terminator thread: premature exit requested - quitting...
   $ echo $?
   1

The same command line as above but with bzip2 -d will return a good
exit status.

Ulrich

[1] https://bugs.gentoo.org/show_bug.cgi?id=417657#c17



Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-27 Thread Florian Philipp
Am 27.09.2012 09:22, schrieb Florian Philipp:
 Am 26.09.2012 23:53, schrieb Michael Mol:
 On Wed, Sep 26, 2012 at 5:27 PM, Florian Philipp li...@binarywings.net 
 wrote:
 Am 26.09.2012 22:43, schrieb Matt Turner:
 On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol mike...@gmail.com wrote:
 A few months ago, I filed bug 423651 to ask that bzip2 on the install
 media be replaced with
  pbzip2. It was closed a short while later, telling me that it'd
 involve changing what's kept in @system, and that had to be discussed
 here, rather than in a bug report.

 If we're going to ship a parallel bzip2 implementation, it should be
 lbzip2 and not pbzip2.

 lbzip2 can decompress bz2 archives with multiple threads that haven't
 been compressed with lbzip2/pbzip2.


 This seems relevant, especially comment 12ff:
 https://bugs.gentoo.org/show_bug.cgi?id=309683

 For further anecdotal evidence: I've used pbzip2 with USE=symlink for
 several months now and never had trouble with it. Checking out lbzip2
 now. I noticed it doesn't install a bunzip2 symlink.

 Piotr Szymaniak asked me about lbzip2, and I bounced the question over
 to my friend. He didn't investigate it deeply; it crashed (OOM or
 something else, I don't know) when he tried it on a large file. Could
 have been from 2GB to 2TB, from what he has laying around. I don't
 know; I didn't get that one in writing. :)

 But if it proves to be stable for small and very large files, I'd have
 no complaint. :)

 
 I just encountered this:
 
 bzip2 -c /srv/qemu/hpwin.img /dev/null
 bzip2:
 /var/tmp/portage/app-arch/lbzip2-2.2/work/lbzip2-2.2/src/encode.c:794:
 generate_initial_trees: Assertion `a  b' failed.
 
 Something in that file is upsetting lbzip2. I'm investigating.

Okay, reported and (hopefully) fixed in
https://bugs.gentoo.org/show_bug.cgi?id=436382

In my infinite confidence in my own coding and testing skills I suggest
copying the proposed patch to /etc/portage/patches/app-arch/lbzip2-2.2
before trying out lbzip2. ;-)

Regards,
Florian Philipp



signature.asc
Description: OpenPGP digital signature


[gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-26 Thread Michael Mol
A few months ago, I filed bug 423651 to ask that bzip2 on the install
media be replaced with
 pbzip2. It was closed a short while later, telling me that it'd
involve changing what's kept in @system, and that had to be discussed
here, rather than in a bug report.

Here's a detailed description of how pbzip2 operates, as described by
a friend of mine:

 pbzip2's compression routine splits the input into blocks (with a default of 
 900,000
 bytes), which it then feeds into the standard bzip2 compression routine. The 
 output
 of the various calls to the bzip2 compression routine are then concatenated 
 together.
 The end result is the same as if you had first used the split command on 
 the input,
 run individual bzip2 commands on the split pieces, then recombined the 
 individual
 bz2 files using cat.

 The down side to this is that you have multiple file headers, footers, and 
 byte-align
 padding, plus the fact that bzip2 does a RLE compression stage to fill the 
 buffer it
 feeds to the BWT, the main part of the compression routine. If you happen to 
 have a
 section with 1MiB of the same byte, the pbzip2 front-end will split that into 
 two blocks
 (at the default settings) and feed them to separate bzip2 compressors. bzip2 
 will
 then compress the first block down to a buffer of about 17kiB before passing 
 it on
 to be compressed further, and the rest of the data would have fit within this 
 block, if
 pbzip2 hadn't split it the way it had.

 As for decompression, pbzip2 can only really do parallel decompression of 
 files that it
 created, since it seeks for the bz2 file header in order to split it to 
 different workers. One
 reason for this is that the bz2 block header is not byte aligned.

I really don't know how to carry this discussion any further than
this; I'll answer any questions I can.

-- 
:wq



Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-26 Thread Matt Turner
On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol mike...@gmail.com wrote:
 A few months ago, I filed bug 423651 to ask that bzip2 on the install
 media be replaced with
  pbzip2. It was closed a short while later, telling me that it'd
 involve changing what's kept in @system, and that had to be discussed
 here, rather than in a bug report.

If we're going to ship a parallel bzip2 implementation, it should be
lbzip2 and not pbzip2.

lbzip2 can decompress bz2 archives with multiple threads that haven't
been compressed with lbzip2/pbzip2.



Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-26 Thread Florian Philipp
Am 26.09.2012 22:43, schrieb Matt Turner:
 On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol mike...@gmail.com wrote:
 A few months ago, I filed bug 423651 to ask that bzip2 on the install
 media be replaced with
  pbzip2. It was closed a short while later, telling me that it'd
 involve changing what's kept in @system, and that had to be discussed
 here, rather than in a bug report.
 
 If we're going to ship a parallel bzip2 implementation, it should be
 lbzip2 and not pbzip2.
 
 lbzip2 can decompress bz2 archives with multiple threads that haven't
 been compressed with lbzip2/pbzip2.
 

This seems relevant, especially comment 12ff:
https://bugs.gentoo.org/show_bug.cgi?id=309683

For further anecdotal evidence: I've used pbzip2 with USE=symlink for
several months now and never had trouble with it. Checking out lbzip2
now. I noticed it doesn't install a bunzip2 symlink.

Regards,
Florian Philipp



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-26 Thread Chí-Thanh Christopher Nguyễn
Michael Mol schrieb:
 A few months ago, I filed bug 423651 to ask that bzip2 on the install
 media be replaced with
  pbzip2.

If I understand correctly, pbzip2 depends on bzip2. So what you are
asking is that pbzip2 is preferred over bzip2 when both are installed,
and that pbzip2 is installed by default?

I have so far encountered only one anecdotal case in #gentoo IRC where
pbzip2[symlink] caused problems in emerging a package. Disabling the
symlink flag made the problem go away. However I can't point to the
report right now, maybe someone with searchable backlog can uncover it.

A different question is whether in the cases where parallel bzip2 makes
sense, is it really the best solution? xz is outperforming bzip2's
compression ratio for large files (for an informal comparison, see bug
434350). And xz is faster at decompression, which offsets the parallel
advantage to some degree.


Best regards,
Chí-Thanh Christopher Nguyễn




Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-26 Thread Michael Mol
On Wed, Sep 26, 2012 at 5:27 PM, Florian Philipp li...@binarywings.net wrote:
 Am 26.09.2012 22:43, schrieb Matt Turner:
 On Wed, Sep 26, 2012 at 1:30 PM, Michael Mol mike...@gmail.com wrote:
 A few months ago, I filed bug 423651 to ask that bzip2 on the install
 media be replaced with
  pbzip2. It was closed a short while later, telling me that it'd
 involve changing what's kept in @system, and that had to be discussed
 here, rather than in a bug report.

 If we're going to ship a parallel bzip2 implementation, it should be
 lbzip2 and not pbzip2.

 lbzip2 can decompress bz2 archives with multiple threads that haven't
 been compressed with lbzip2/pbzip2.


 This seems relevant, especially comment 12ff:
 https://bugs.gentoo.org/show_bug.cgi?id=309683

 For further anecdotal evidence: I've used pbzip2 with USE=symlink for
 several months now and never had trouble with it. Checking out lbzip2
 now. I noticed it doesn't install a bunzip2 symlink.

Piotr Szymaniak asked me about lbzip2, and I bounced the question over
to my friend. He didn't investigate it deeply; it crashed (OOM or
something else, I don't know) when he tried it on a large file. Could
have been from 2GB to 2TB, from what he has laying around. I don't
know; I didn't get that one in writing. :)

But if it proves to be stable for small and very large files, I'd have
no complaint. :)

-- 
:wq



Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-26 Thread Michael Mol
On Wed, Sep 26, 2012 at 5:49 PM, Chí-Thanh Christopher Nguyễn
chith...@gentoo.org wrote:
 Michael Mol schrieb:
 A few months ago, I filed bug 423651 to ask that bzip2 on the install
 media be replaced with
  pbzip2.

 If I understand correctly, pbzip2 depends on bzip2. So what you are
 asking is that pbzip2 is preferred over bzip2 when both are installed,
 and that pbzip2 is installed by default?

pbzip2 uses libbzip2, which I understand bzip2 to also be a wrapper around.


 I have so far encountered only one anecdotal case in #gentoo IRC where
 pbzip2[symlink] caused problems in emerging a package. Disabling the
 symlink flag made the problem go away. However I can't point to the
 report right now, maybe someone with searchable backlog can uncover it.

pbzip2[symlink] is more or less the scenario I'd like to see.


 A different question is whether in the cases where parallel bzip2 makes
 sense, is it really the best solution? xz is outperforming bzip2's
 compression ratio for large files (for an informal comparison, see bug
 434350). And xz is faster at decompression, which offsets the parallel
 advantage to some degree.

xz is faster for decompression, by my inspiration case was during
system installation, so compression. Last I looked, xz was still very
slow for threaded compression. And it's not block-oriented, so I don't
think that's really possible without loss of compression efficiency,
anyway...and my use cases range from 4-8 physical cores.

-- 
:wq



Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-26 Thread Mike Gilbert
On Wed, Sep 26, 2012 at 5:59 PM, Michael Mol mike...@gmail.com wrote:
 On Wed, Sep 26, 2012 at 5:49 PM, Chí-Thanh Christopher Nguyễn
 chith...@gentoo.org wrote:
 Michael Mol schrieb:
 A few months ago, I filed bug 423651 to ask that bzip2 on the install
 media be replaced with
  pbzip2.

 If I understand correctly, pbzip2 depends on bzip2. So what you are
 asking is that pbzip2 is preferred over bzip2 when both are installed,
 and that pbzip2 is installed by default?

 pbzip2 uses libbzip2, which I understand bzip2 to also be a wrapper around.


libbz2 is built and installed by the app-arch/bzip2 package. Thus,
app-arch/pbzip2 depends on app-arch/bzip2, unless someone rips libbz2
out into a separate ebuild.



Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-26 Thread Christoph Junghans
2012/9/26 Mike Gilbert flop...@gentoo.org:
 On Wed, Sep 26, 2012 at 5:59 PM, Michael Mol mike...@gmail.com wrote:
 On Wed, Sep 26, 2012 at 5:49 PM, Chí-Thanh Christopher Nguyễn
 chith...@gentoo.org wrote:
 Michael Mol schrieb:
 A few months ago, I filed bug 423651 to ask that bzip2 on the install
 media be replaced with
  pbzip2.

 If I understand correctly, pbzip2 depends on bzip2. So what you are
 asking is that pbzip2 is preferred over bzip2 when both are installed,
 and that pbzip2 is installed by default?

 pbzip2 uses libbzip2, which I understand bzip2 to also be a wrapper around.


 libbz2 is built and installed by the app-arch/bzip2 package. Thus,
 app-arch/pbzip2 depends on app-arch/bzip2, unless someone rips libbz2
 out into a separate ebuild.
That sound like a plan. Maybe bzip2 should become a virtual as busybox
also provides an implementation.



-- 
Christoph Junghans
http://dev.gentoo.org/~ottxor/



Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-26 Thread Michael Mol
On Wed, Sep 26, 2012 at 6:57 PM, Christoph Junghans ott...@gentoo.org wrote:
 2012/9/26 Mike Gilbert flop...@gentoo.org:
 On Wed, Sep 26, 2012 at 5:59 PM, Michael Mol mike...@gmail.com wrote:
 On Wed, Sep 26, 2012 at 5:49 PM, Chí-Thanh Christopher Nguyễn
 chith...@gentoo.org wrote:
 Michael Mol schrieb:
 A few months ago, I filed bug 423651 to ask that bzip2 on the install
 media be replaced with
  pbzip2.

 If I understand correctly, pbzip2 depends on bzip2. So what you are
 asking is that pbzip2 is preferred over bzip2 when both are installed,
 and that pbzip2 is installed by default?

 pbzip2 uses libbzip2, which I understand bzip2 to also be a wrapper around.


 libbz2 is built and installed by the app-arch/bzip2 package. Thus,
 app-arch/pbzip2 depends on app-arch/bzip2, unless someone rips libbz2
 out into a separate ebuild.
 That sound like a plan. Maybe bzip2 should become a virtual as busybox
 also provides an implementation.

This makes sense. And going back to my initial issue, I don't really
care which implementation gets used on the bootable media, so long as
it supports scaling to use my CPU cores.

-- 
:wq



Re: [gentoo-dev] ship app-arch/pbzip2 instead of app-arch/bzip2

2012-09-26 Thread Diego Elio Pettenò
On 26/09/2012 15:57, Christoph Junghans wrote:
 That sound like a plan. Maybe bzip2 should become a virtual as busybox
 also provides an implementation.

No, just, no.

-- 
Diego Elio Pettenò — Flameeyes
flamee...@flameeyes.eu — http://blog.flameeyes.eu/