Re: Propose Release Goals (delayed ;) - xz compression
Off list. Thanks! Richard
Re: Propose Release Goals (delayed ;) - xz compression
On Wed, 16 Oct 2013 19:49:54 +0200 Dominik George n...@naturalnet.de wrote: Hi, The only problem is that on small machines (things like the BeagleBone) xz compression requires enough memory that you have to enable swap to use dpkg. Now on a machine with a sensible disk this is not a problem, but on a machine where the disk is an SD-card it is a disaster. correct me if I'm wrong, but it appears to me that xz compression has become the default in dpkg. With that in mind, won't this issue come up anyway? I mean, once a maintainer fixes a bug in a pckage and uplods it, the binries ill get xz compression. So this issue is in no way specific to making it a release goal. It's not only memory footprint. xz at default preset is more then 10 times slower when compressing then gzip and file is smaller 20%. signature.asc Description: PGP signature
Re: Propose Release Goals (delayed ;) - xz compression
On Fri, Oct 25, 2013 at 03:33:42PM +0200, Marko Randjelovic wrote: correct me if I'm wrong, but it appears to me that xz compression has become the default in dpkg. With that in mind, won't this issue come up anyway? I mean, once a maintainer fixes a bug in a pckage and uplods it, the binries ill get xz compression. So this issue is in no way specific to making it a release goal. It's not only memory footprint. xz at default preset is more then 10 times slower when compressing then gzip and file is smaller 20%. xz has slow compression, fast decompression. You're not really going to build packages on any box where compression speed is a blocker, and even if you do, actually building the package will take a wolf share of the time. On the other hand, quite a few users will _decompress_ stuff on terribly underpowered boxes, and xz shines there. -- ᛊᚨᚾᛁᛏᚣ᛫ᛁᛊ᛫ᚠᛟᚱ᛫ᚦᛖ᛫ᚹᛖᚨᚲ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131025135238.ga18...@angband.pl
Re: Propose Release Goals (delayed ;) - xz compression
On Fri, 25 Oct 2013 15:52:38 +0200 Adam Borowski kilob...@angband.pl wrote: xz has slow compression, fast decompression. You're not really going to build packages on any box where compression speed is a blocker, and even if you do, actually building the package will take a wolf share of the time. On the other hand, quite a few users will _decompress_ stuff on terribly underpowered boxes, and xz shines there. Not quite, xz is also slower than gzip in decompression, cca 3 times, which is not neglectable on slow machines, especially when installing large sets of packages. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131025173608.3dc4e...@eunet.rs
Re: Propose Release Goals (delayed ;) - xz compression
On Fri, Oct 25, 2013 at 05:36:08PM +0200, Marko Randjelovic wrote: Not quite, xz is also slower than gzip in decompression, cca 3 times, which is not neglectable on slow machines, especially when installing large sets of packages. This is incorrect. xz -[012] is way better in terms of decompression time and compression ratio then gzip -9 (the old default). Bastian -- We fight only when there is no other choice. We prefer the ways of peaceful contact. -- Kirk, Spectre of the Gun, stardate 4385.3 -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131025172615.ga6...@mail.waldi.eu.org
Re: Propose Release Goals (delayed ;) - xz compression
On Fri, 18 Oct 2013, Guillem Jover wrote: For example on one of my 64-bit systems, with 220481 paths installed, I go from 62.8 MiB to 46.1 MiB max resident memory, a saving of 16.7 MiB. That should compensate a bit for the slight increase in memory usage from xz. This is great, thank you! -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131019140757.gd18...@khazad-dum.debian.net
Re: Propose Release Goals (delayed ;) - xz compression
On Wed, October 16, 2013 16:20, Hideki Yamane wrote: As dpkg introduced xz compression by default, we can make whole packages xz-ed now. I think it's worth to try, so propose it as a release goal (I know it should be sent before its dead line, but please read). Because dpkg =1.17.0 already does this by default, we can assume that the vast majority of packages will automatically use xz compression by the simple fact that they've been uploaded during the jessie cycle. What would the release goal add to this? Do you propose e.g. binNMU's of those packages that haven't been uploaded since dpkg 1.17 entered the archive? If so, at what point would you start doing that, and how do you deal with arch:all packages? Please make the proposed actions more explicit. Thijs -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/151e1063bb6abc6b7748cd22a95fbd46.squir...@aphrodite.kinkhorst.nl
Re: Propose Release Goals (delayed ;) - xz compression
David Goodenough david.goodenough at btconnect.com writes: xy may only use a tiny bit, but the combination of apt-get, dpkg and xy seems to cause problems. Its not just BeagleBones, there are x86 machines with just 64MB still on sale. SOL then. It’s actually apt/dpkg that takes that much memory because, you know, a database listing 3 binary packages in sid *does* take quite some RAM. We have the same problem on m68k, but you can’t do much against that (except, possibly, use a more memory-efficient internal representation in those tools). So, this means that, yes, you need a total of at least 128 MiB RAM+swap, if not more, to use apt/dpkg in sid (and recent releases were not much smaller). xz decompression is a nōn-issue on virtually all Debian systems. xz compression, now, may be an issue, but I still support it as long as the compression level is no larger than -6 (-7 for debug packages, should that be desired), and -e is only used in extreme cases (pun intended). (I have other issues with xz adoption, but they appear to not be an issue in Debian so I skip them here.) bye, //mirabilos ObCaptcha: unhelpful -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/loom.20131017t132726-...@post.gmane.org
Re: Propose Release Goals (delayed ;) - xz compression
On Thu, Oct 17, 2013 at 11:31:23AM +, Thorsten Glaser wrote: So, this means that, yes, you need a total of at least 128 MiB RAM+swap, if not more, to use apt/dpkg in sid (and recent releases were not much smaller). Managed with ~100M with squeeze (in VMs) — I remember because I recall yum failing on the same size. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131017114327.gb7...@bryant.redmars.org
skipping bioinformatics on some architectures ? (was Re: Propose Release Goals (delayed ;) - xz compression)
Le Thu, Oct 17, 2013 at 11:31:23AM +, Thorsten Glaser a écrit : It’s actually apt/dpkg that takes that much memory because, you know, a database listing 3 binary packages in sid *does* take quite some RAM. We have the same problem on m68k, but you can’t do much against that (except, possibly, use a more memory-efficient internal representation in those tools). Hi Thorsten, that seems to argue for reducing the list of packages in m68k or other ports. Some programs are obviously useless on low-power platforms, for instance, most of the packages with the Field::Biology tag. If you are interested I would be willing to inspect the corner cases (such as mencal ?), in order to devise a better filter for listing the packages related to biology and bioinformatics to skip on m68k, and other low-power architectures if their porters are intersted. Cheers, -- Charles Plessy Debian Med packaging team, http://www.debian.org/devel/debian-med Tsurumi, Kanagawa, Japan -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131017123934.gi17...@falafel.plessy.net
Re: Propose Release Goals (delayed ;) - xz compression
SEE 271...@bugs.debian.org Maybe insted of reading the file in memory concatenating then mmaping the resulting file will help in case of low memory Bastien Le 17 oct. 2013 13:43, Jonathan Dowland j...@debian.org a écrit : On Thu, Oct 17, 2013 at 11:31:23AM +, Thorsten Glaser wrote: So, this means that, yes, you need a total of at least 128 MiB RAM+swap, if not more, to use apt/dpkg in sid (and recent releases were not much smaller). Managed with ~100M with squeeze (in VMs) — I remember because I recall yum failing on the same size. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131017114327.gb7...@bryant.redmars.org
Re: skipping bioinformatics on some architectures ? (was Re: Propose Release Goals (delayed ;) - xz compression)
On Thu, Oct 17, 2013 at 09:39:34PM +0900, Charles Plessy wrote: Le Thu, Oct 17, 2013 at 11:31:23AM +, Thorsten Glaser a écrit : It’s actually apt/dpkg that takes that much memory because, you know, a database listing 3 binary packages in sid *does* take quite some RAM. We have the same problem on m68k, but you can’t do much against that (except, possibly, use a more memory-efficient internal representation in those tools). Hi Thorsten, that seems to argue for reducing the list of packages in m68k or other ports. Some programs are obviously useless on low-power platforms, for instance, most of the packages with the Field::Biology tag. If you are interested I would be willing to inspect the corner cases (such as mencal ?), in order to devise a better filter for listing the packages related to biology and bioinformatics to skip on m68k, and other low-power architectures if their porters are intersted. I see your good intention in doing this to help embedded-ish architectures, but you have to keep in mind that doing this at the package level does not scale. This use case needs to be solved at the archive level, like emdebian does. -- Antonio Terceiro terce...@debian.org signature.asc Description: Digital signature
Re: Propose Release Goals (delayed ;) - xz compression
Hi! On Wed, 2013-10-16 at 17:32:37 +0100, David Goodenough wrote: xy may only use a tiny bit, but the combination of apt-get, dpkg and xy seems to cause problems. Its not just BeagleBones, there are x86 machines with just 64MB still on sale. Ok, I went through the dpkg code, and have reduced a bit its memory usage (and line count too!), which will get included in 1.17.2. First, when loading the .list files, dpkg was slurping them into a non-freeing memory pool, and populating the file list structures from that, the problem is that all the shared files and directories were left allocated on the heap; now the code just allocates the paths that are non-duped. And second, for *each* non-shared file on the system it was wasting 9 pointers, which is pretty significant, more so when taking into account 64-bit pointers. For example on one of my 64-bit systems, with 220481 paths installed, I go from 62.8 MiB to 46.1 MiB max resident memory, a saving of 16.7 MiB. That should compensate a bit for the slight increase in memory usage from xz. Thanks, Guillem -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131018052957.ga27...@gaara.hadrons.org
Propose Release Goals (delayed ;) - xz compression
Hi,
As dpkg introduced xz compression by default, we can make whole
packages xz-ed now. I think it's worth to try, so propose it as
a release goal (I know it should be sent before its dead line, but
please read).
-
item)
rebuild whole package with xz
Benefit)
users : it reduces download (update) time.
mirror admins : it cuts traffic.
How to archive it? (a.k.a. costs))
Just rebuild your package with dpkg (=1.17.0), cheap.
(it's not in jessie yet, Bug#717983 prevents to migrate to testing,
but now fixed in git and pending).
How to check it?)
Yes, just ar -x and you'll see it as data.tar.{gz,bz2,xz}, easy.
Trackable?)
So yes, just do it as above for whole repository and track numbers
and list non-xz-ed package.
-
Any idea for this?
It's worth to try *and* not so difficult, IMHO.
--
Hideki Yamane
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
http://lists.debian.org/CAPpVEmWoy0_Nx2dH7zuyCRAFaQU6bYGeNUXga8=kddcvnov...@mail.gmail.com
Re: Propose Release Goals (delayed ;) - xz compression
On Wednesday 16 Oct 2013, Hideki Yamane wrote:
Hi,
As dpkg introduced xz compression by default, we can make whole
packages xz-ed now. I think it's worth to try, so propose it as
a release goal (I know it should be sent before its dead line, but
please read).
---
-- item)
rebuild whole package with xz
Benefit)
users : it reduces download (update) time.
mirror admins : it cuts traffic.
How to archive it? (a.k.a. costs))
Just rebuild your package with dpkg (=1.17.0), cheap.
(it's not in jessie yet, Bug#717983 prevents to migrate to testing,
but now fixed in git and pending).
How to check it?)
Yes, just ar -x and you'll see it as data.tar.{gz,bz2,xz}, easy.
Trackable?)
So yes, just do it as above for whole repository and track numbers
and list non-xz-ed package.
---
--
Any idea for this?
It's worth to try *and* not so difficult, IMHO.
The only problem is that on small machines (things like the BeagleBone)
xz compression requires enough memory that you have to enable swap to
use dpkg. Now on a machine with a sensible disk this is not a problem,
but on a machine where the disk is an SD-card it is a disaster.
David
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive:
http://lists.debian.org/201310161536.55485.david.goodeno...@btconnect.com
Re: Propose Release Goals (delayed ;) - xz compression
David Goodenough david.goodeno...@btconnect.com writes: The only problem is that on small machines (things like the BeagleBone) xz compression requires enough memory that you have to enable swap to use dpkg. Now on a machine with a sensible disk this is not a problem, but on a machine where the disk is an SD-card it is a disaster. From the xz manpage: # Preset DictSize CompCPU CompMem DecMem # -0 256 KiB 03 MiB1 MiB # -1 1 MiB 19 MiB2 MiB # -2 2 MiB 2 17 MiB3 MiB # -3 4 MiB 3 32 MiB5 MiB # -4 4 MiB 4 48 MiB5 MiB # -5 8 MiB 5 94 MiB9 MiB # -6 8 MiB 6 94 MiB9 MiB # -7 16 MiB 6 186 MiB 17 MiB # -8 32 MiB 6 370 MiB 33 MiB # -9 64 MiB 6 674 MiB 65 MiB At the default preset (-6), the required RAM for decompressing is about 9MB. The BeagleBone seems to have 256MB of memory (that's what Wikipedia says), so 9MB shouldn't be an issue. And if 9MB is too much for some random board, xz -0 still compresses better than gzip -9 (or so it should) with only 1MB of DecMem. -- Marius Gavrilescu pgpWPbktNjNTF.pgp Description: PGP signature
Re: Propose Release Goals (delayed ;) - xz compression
On Wednesday 16 Oct 2013, Marius Gavrilescu wrote: David Goodenough david.goodeno...@btconnect.com writes: The only problem is that on small machines (things like the BeagleBone) xz compression requires enough memory that you have to enable swap to use dpkg. Now on a machine with a sensible disk this is not a problem, but on a machine where the disk is an SD-card it is a disaster. From the xz manpage: # Preset DictSize CompCPU CompMem DecMem # -0 256 KiB 03 MiB1 MiB # -1 1 MiB 19 MiB2 MiB # -2 2 MiB 2 17 MiB3 MiB # -3 4 MiB 3 32 MiB5 MiB # -4 4 MiB 4 48 MiB5 MiB # -5 8 MiB 5 94 MiB9 MiB # -6 8 MiB 6 94 MiB9 MiB # -7 16 MiB 6 186 MiB 17 MiB # -8 32 MiB 6 370 MiB 33 MiB # -9 64 MiB 6 674 MiB 65 MiB At the default preset (-6), the required RAM for decompressing is about 9MB. The BeagleBone seems to have 256MB of memory (that's what Wikipedia says), so 9MB shouldn't be an issue. And if 9MB is too much for some random board, xz -0 still compresses better than gzip -9 (or so it should) with only 1MB of DecMem. xy may only use a tiny bit, but the combination of apt-get, dpkg and xy seems to cause problems. Its not just BeagleBones, there are x86 machines with just 64MB still on sale. David -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201310161732.38040.david.goodeno...@btconnect.com
Re: Propose Release Goals (delayed ;) - xz compression
On Wed, Oct 16, 2013 at 05:32:37PM +0100, David Goodenough wrote: xy may only use a tiny bit, but the combination of apt-get, dpkg and xy seems to cause problems. Its not just BeagleBones, there are x86 machines with just 64MB still on sale. Do we expect to build Debian packages on such systems? -- http://www.cafepress.com/trunktees -- geeky funny T-shirts http://gtdfh.branchable.com/ -- GTD for hackers -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131016163521.gb4...@mavolio.codethink.co.uk
Re: Propose Release Goals (delayed ;) - xz compression
On Wednesday 16 Oct 2013, Lars Wirzenius wrote: On Wed, Oct 16, 2013 at 05:32:37PM +0100, David Goodenough wrote: xy may only use a tiny bit, but the combination of apt-get, dpkg and xy seems to cause problems. Its not just BeagleBones, there are x86 machines with just 64MB still on sale. Do we expect to build Debian packages on such systems? no, but we do expect to install on them. David -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201310161740.50457.david.goodeno...@btconnect.com
Re: Propose Release Goals (delayed ;) - xz compression
Lars Wirzenius l...@liw.fi writes: Do we expect to build Debian packages on such systems? David's point was that installing such a package would require too much memory due to xz's decompression memory requirements (9MB with default options). -- Marius Gavrilescu pgpuiMWwJJsFS.pgp Description: PGP signature
Re: Propose Release Goals (delayed ;) - xz compression
On Wed, Oct 16, 2013 at 07:19:19PM +0300, Marius Gavrilescu wrote: At the default preset (-6), the required RAM for decompressing is about 9MB. The BeagleBone seems to have 256MB of memory (that's what Wikipedia says), so 9MB shouldn't be an issue. Didn't we discuss this last year already? And if 9MB is too much for some random board, xz -0 still compresses better than gzip -9 (or so it should) with only 1MB of DecMem. -2 or -3 should be sane settings for all systems. Bastian -- Those who hate and fight must stop themselves -- otherwise it is not stopped. -- Spock, Day of the Dove, stardate unknown -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20131016172230.ga10...@mail.waldi.eu.org
Re: Propose Release Goals (delayed ;) - xz compression
Hi, The only problem is that on small machines (things like the BeagleBone) xz compression requires enough memory that you have to enable swap to use dpkg. Now on a machine with a sensible disk this is not a problem, but on a machine where the disk is an SD-card it is a disaster. correct me if I'm wrong, but it appears to me that xz compression has become the default in dpkg. With that in mind, won't this issue come up anyway? I mean, once a maintainer fixes a bug in a pckage and uplods it, the binries ill get xz compression. So this issue is in no way specific to making it a release goal. On the contrary, if we do it right now for the whole archive, users have something to rely on and, should it really be an issue, won't see their systems slowly failing more and more over time. That said, if choosing xz as default compression is harmful, then this harm has already been done and should have been prevented before making xz default in dpkg 1.17.0. I, for one, consider the release goal a good idea as for stable users, it provides a reliable moment in time when things will break so they can prepare for it. Still, I also think that in the first place, things *won't* break, as stated by others. Cheers, Nik -- * concerning Mozilla code leaking assertion failures to tty without D-BUS * mirabilos That means, D-BUS is a tool that makes software look better than it actually is. PGP-Fingerprint: 3C9D 54A4 7575 C026 FB17 FD26 B79A 3C16 A0C4 F296 signature.asc Description: Digital signature
Re: Propose Release Goals (delayed ;) - xz compression
On 10/17/2013 12:35 AM, Lars Wirzenius wrote: On Wed, Oct 16, 2013 at 05:32:37PM +0100, David Goodenough wrote: xy may only use a tiny bit, but the combination of apt-get, dpkg and xy seems to cause problems. Its not just BeagleBones, there are x86 machines with just 64MB still on sale. Do we expect to build Debian packages on such systems? That's not enough RAM for uncompromising the default initrd and booting from it. And it's been years this way. So IMO, no. Could we *not* have the same topics in loop in this list, and move on? Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/525ed243.7080...@debian.org
