Re: packages compressed with xz
On Sat, Dec 04, 2010 at 11:19:21PM +0100, Simon L. B. Nielsen wrote: On 30 Nov 2010, at 03:16, jhell wrote: Agreed. Soon can be quantified by actual need and of which there is not much need except for larger packages but adding this would just add unneeded complication to the system that is already in place. We are running out of diskspace on event the FTP master site - currently we are at ~1TB. The xz compression gives as significant space saving - so there is already a need. PS. anyone saying a 1 TB etc. disk is cheap will be ignored. And all ftp mirror admins will be very grateful to sync smaller .xz packages. :) pgpADHhbR7zTC.pgp Description: PGP signature
Re: packages compressed with xz
On Fri, 03 Dec 2010 23:07:51 +0200 Volodymyr Kostyrko c.kw...@gmail.com wrote: 30.11.2010 04:40, Julien Laffaye wrote: You can specify limits during compression, so the question is should we do that so that hosts with N MB of RAM can decompress packages? Do we retain the compression ratio over bzip2 if we limit compression memory to 512 MB so that decompression would be possible with, say, 128 MB? According to xz(1), in its default mode (-6), xz uses ~100MiB for compression and ~10MiB for decompression. That seems to be acceptable. You possibly miss something about compression/decompression. The designated memory size is not directly affected only by compression mode. When decompressing you will need memory for: 1. Data history. 2. Dictionary. 3. Some indexes. And those ones are all empty at start. So say, if you are compressing something really huge trying to use 4G of memory you end using that much memory between 2G - 3G of source data. And we will need 512MB to decompress that hunk of data. Are the packages _that_ large? [ .. ] The biggest package that can be produced by a port it's a bit over 10G. -- IOnut - Un^d^dregistered ;) FreeBSD user Intellectual Property is nowhere near as valuable as Intellect FreeBSD committer - ite...@freebsd.org, PGP Key ID 057E9F8B493A297B signature.asc Description: PGP signature
Re: packages compressed with xz
The biggest package that can be produced by a port it's a bit over 10G. Curious - which one? -- Eitan Adler ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: packages compressed with xz
Eitan Adler writes: The biggest package that can be produced by a port it's a bit over 10G. Curious - which one? OpenOffice(-3)? Robert Huff ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: packages compressed with xz
On Sun, 5 Dec 2010 11:41:29 -0500 Eitan Adler li...@eitanadler.com wrote: Curious - which one? OpenOffice(-3)? He told me on IRC - something in games/ $grep -R NO_PACKAGE /usr/ports/games :-) Hum, looks like either they got smaller or QAT doesn't have all the games packages built yet: -rw-r--r-- 1 root wheel 5.3G Nov 29 02:42 crafty-tablebase-pawn-20070910.tbz -rw-r--r-- 1 root wheel 1.7G Nov 29 01:59 crafty-tablebases-no-pawn-20070910.tbz -- IOnut - Un^d^dregistered ;) FreeBSD user Intellectual Property is nowhere near as valuable as Intellect FreeBSD committer - ite...@freebsd.org, PGP Key ID 057E9F8B493A297B signature.asc Description: PGP signature
Re: packages compressed with xz
05.12.2010 18:03, Ion-Mihai Tetcu wrote: And those ones are all empty at start. So say, if you are compressing something really huge trying to use 4G of memory you end using that much memory between 2G - 3G of source data. And we will need 512MB to decompress that hunk of data. Are the packages _that_ large? [ .. ] The biggest package that can be produced by a port it's a bit over 10G. 0k, I'll try to be sharper. How many of that huge packages will run on a prehistoric cpu/64Mb mem combo? I'll bet even working with the OpenOffice will be... challenging. Let's take the other route. Packages that large will benefit from maximum xz compression most. The question clearly is Would we like to ditch maximum compression for those huge ones taking care of the hardware that would never be ready to run them? Don't listen to me anyway. Our first goal is clearly making FreeBSD work everywhere. Capping the xz limits would do nothing for me personally because all of my machines are capable of building any software I want - I don't use packages at all. So I don't have a right to vote. We should think from the point of people running that hardware. And I think we should support them. -- Sphinx of black quartz judge my vow. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: packages compressed with xz
On 30 Nov 2010, at 03:16, jhell wrote: Agreed. Soon can be quantified by actual need and of which there is not much need except for larger packages but adding this would just add unneeded complication to the system that is already in place. We are running out of diskspace on event the FTP master site - currently we are at ~1TB. The xz compression gives as significant space saving - so there is already a need. PS. anyone saying a 1 TB etc. disk is cheap will be ignored. -- Simon L. B. Nielsen ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: packages compressed with xz
30.11.2010 04:40, Julien Laffaye wrote: You can specify limits during compression, so the question is should we do that so that hosts with N MB of RAM can decompress packages? Do we retain the compression ratio over bzip2 if we limit compression memory to 512 MB so that decompression would be possible with, say, 128 MB? According to xz(1), in its default mode (-6), xz uses ~100MiB for compression and ~10MiB for decompression. That seems to be acceptable. You possibly miss something about compression/decompression. The designated memory size is not directly affected only by compression mode. When decompressing you will need memory for: 1. Data history. 2. Dictionary. 3. Some indexes. And those ones are all empty at start. So say, if you are compressing something really huge trying to use 4G of memory you end using that much memory between 2G - 3G of source data. And we will need 512MB to decompress that hunk of data. Are the packages _that_ large? I think that in worst possible case we need to double the size of package for comfort decompression. If the lower point is 64Mb then 32Mb package compressed with any compression strategy would be decompressed without hitting a swap. We'll need someone to do actual testing but look at this one: # ls -la ImageMagick-6.6.5-10.tar.xz -rw-r--r-- 1 root wheel 6316324 21 ноя 23:52 ImageMagick-6.6.5-10.tar.xz # time xz -dt ImageMagick-6.6.5-10.tar.xz 0.860u 0.036s 0:01.02 87.2% 68+1492k 56+0io 1pf+0w -- Sphinx of black quartz judge my vow. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: packages compressed with xz
Ion-Mihai Tetcu ite...@freebsd.org wrote: It would be nice to support xz(1) compression for large selective packages like firefox or openoffice as those will never run on smaller systems. Trouble is it ain't no way (CPU, space, banhdwidth on our side and space,bandwidth on our mirrors side) we could build a double set of packages. but perhaps it would be possible to support compressing certain ports' packages using xz _instead of_ bzip2, the specification being made in the port's Makefile and defaulting to bzip2 (current behavior) if no alternative compression method is specified. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: packages compressed with xz
Am 28.11.2010 22:12, schrieb Goran Tal: Now that the base system supports xz compression, it should be used as the default compression for packages. Files compressed with xz are smaller and decompress faster than those compressed with bzip2. This can make an installation much quicker, especially when the complete system is installed or upgraded. Any reasons against it? xz compressed files can take up CONSIDERABLY more memory to decompress than files compressed with bzip2 or gzip. Keep that in mind so that systems that are low on memory can still decompress xz packages. If you don't fit into RAM for decompression, it will be unusable. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: packages compressed with xz
On 11/29/2010 06:24, Matthias Andree wrote: Am 28.11.2010 22:12, schrieb Goran Tal: Now that the base system supports xz compression, it should be used as the default compression for packages. Files compressed with xz are smaller and decompress faster than those compressed with bzip2. This can make an installation much quicker, especially when the complete system is installed or upgraded. Any reasons against it? xz compressed files can take up CONSIDERABLY more memory to decompress than files compressed with bzip2 or gzip. Keep that in mind so that systems that are low on memory can still decompress xz packages. If you don't fit into RAM for decompression, it will be unusable. Adding to this, as the manual says... The decompressing host will need to have at minimal 5% - 20% of memory 'available' for decompression of what the compressing host had. Seeing as FreeBSD still runs on systems with memory as little as 200MB ~20% of 1024MB and quite possible to run on systems with memory of 64MB ~5% of 1024MB I would not see any benefit in modifying the default memory limit on a compressing host to accommodate for these system rather than using gzip(1) or bzip2(1) by default. It would be nice to support xz(1) compression for large selective packages like firefox or openoffice as those will never run on smaller systems. -- jhell,v ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: packages compressed with xz
On Mon, 29 Nov 2010 13:31:10 -0500 jhell jh...@dataix.net wrote: On 11/29/2010 06:24, Matthias Andree wrote: Am 28.11.2010 22:12, schrieb Goran Tal: Now that the base system supports xz compression, it should be used as the default compression for packages. Files compressed with xz are smaller and decompress faster than those compressed with bzip2. This can make an installation much quicker, especially when the complete system is installed or upgraded. Any reasons against it? xz compressed files can take up CONSIDERABLY more memory to decompress than files compressed with bzip2 or gzip. Keep that in mind so that systems that are low on memory can still decompress xz packages. If you don't fit into RAM for decompression, it will be unusable. Adding to this, as the manual says... The decompressing host will need to have at minimal 5% - 20% of memory 'available' for decompression of what the compressing host had. Seeing as FreeBSD still runs on systems with memory as little as 200MB ~20% of 1024MB and quite possible to run on systems with memory of 64MB ~5% of 1024MB I would not see any benefit in modifying the default memory limit on a compressing host to accommodate for these system rather than using gzip(1) or bzip2(1) by default. It would be nice to support xz(1) compression for large selective packages like firefox or openoffice as those will never run on smaller systems. Trouble is it ain't no way (CPU, space, banhdwidth on our side and space,bandwidth on our mirrors side) we could build a double set of packages. -- IOnut - Un^d^dregistered ;) FreeBSD user Intellectual Property is nowhere near as valuable as Intellect FreeBSD committer - ite...@freebsd.org, PGP Key ID 057E9F8B493A297B signature.asc Description: PGP signature
Re: packages compressed with xz
Am 29.11.2010 19:31, schrieb jhell: Adding to this, as the manual says... The decompressing host will need to have at minimal 5% - 20% of memory 'available' for decompression of what the compressing host had. Seeing as FreeBSD still runs on systems with memory as little as 200MB ~20% of 1024MB and quite possible to run on systems with memory of 64MB ~5% of 1024MB I would not see any benefit in modifying the default memory limit on a compressing host to accommodate for these system rather than using gzip(1) or bzip2(1) by default. You can specify limits during compression, so the question is should we do that so that hosts with N MB of RAM can decompress packages? Do we retain the compression ratio over bzip2 if we limit compression memory to 512 MB so that decompression would be possible with, say, 128 MB? It would be nice to support xz(1) compression for large selective packages like firefox or openoffice as those will never run on smaller systems. Yes, would be nice. I doubt it will happen soon. -- Matthias Andree ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: packages compressed with xz
On Tue, Nov 30, 2010 at 12:40:33AM +0100, Matthias Andree wrote: Yes, would be nice. I doubt it will happen soon. It's actually being looked at. As part of the extensive rework of the pointyhat scripts I did this summer, I attempted to factor out all the magic constants, including the definition of PKGSUFFIX. There are some other bugs that need to be fixed (having to deal with INDEX and duds) before we can get to this, however. mcl ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: packages compressed with xz
On 11/29/2010 18:40, Matthias Andree wrote: Am 29.11.2010 19:31, schrieb jhell: Adding to this, as the manual says... The decompressing host will need to have at minimal 5% - 20% of memory 'available' for decompression of what the compressing host had. Seeing as FreeBSD still runs on systems with memory as little as 200MB ~20% of 1024MB and quite possible to run on systems with memory of 64MB ~5% of 1024MB I would not see any benefit in modifying the default memory limit on a compressing host to accommodate for these system rather than using gzip(1) or bzip2(1) by default. You can specify limits during compression, so the question is should we do that so that hosts with N MB of RAM can decompress packages? Do we retain the compression ratio over bzip2 if we limit compression memory to 512 MB so that decompression would be possible with, say, 128 MB? Hosts that have [N]MB of free or available memory. Most systems in this case will not have a whole lot of RAM available in any case as they are likely to be utilizing it to its upper most potential. Doing such limiting on the compressors part I would think, be more of a waste of resources as such can be achieved by default just sticking with bzip2(1). Besides, limiting memory to 512MB to what ? shave an ~ small percentage off the top of a resulting package. It would be nice to support xz(1) compression for large selective packages like firefox or openoffice as those will never run on smaller systems. Yes, would be nice. I doubt it will happen soon. Agreed. Soon can be quantified by actual need and of which there is not much need except for larger packages but adding this would just add unneeded complication to the system that is already in place. ~ JMO -- jhell,v ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: packages compressed with xz
On 11/29/2010 23:40, Matthias Andree wrote: You can specify limits during compression, so the question is should we do that so that hosts with N MB of RAM can decompress packages? Do we retain the compression ratio over bzip2 if we limit compression memory to 512 MB so that decompression would be possible with, say, 128 MB? According to xz(1), in its default mode (-6), xz uses ~100MiB for compression and ~10MiB for decompression. That seems to be acceptable. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
packages compressed with xz
Now that the base system supports xz compression, it should be used as the default compression for packages. Files compressed with xz are smaller and decompress faster than those compressed with bzip2. This can make an installation much quicker, especially when the complete system is installed or upgraded. Any reasons against it? ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: packages compressed with xz
Hello, On 28/11/2010 22:12, Goran Tal wrote: Now that the base system supports xz compression, it should be used as the default compression for packages. Files compressed with xz are smaller and decompress faster than those compressed with bzip2. This can make an installation much quicker, especially when the complete system is installed or upgraded. Any reasons against it? I think it's already in CURRENT. On my notebook xz compression and decompression are much slower than any of the other algorithms supported by tar. It has the best compression ratio, though. ___ freebsd-ports@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org
Re: packages compressed with xz
On Sun, 28 Nov 2010 16:12:48 -0500
Goran Tal goran@gmail.com wrote:
Now that the base system supports xz compression, it should be used as
the default compression for packages.
Files compressed with xz are smaller and decompress faster than those
compressed with bzip2. This can make an installation much quicker,
especially when the complete system is installed or upgraded.
Any reasons against it?
Sigh. We enabled that in HEAD, and after some testing and adapting
{base_OS, ports, tools, etc.} to support that, it will probably become
the default.
One thing against it is the major increase in package time (up to 6x,
but by no means this is a show-stopper).
Resulting packages would be, in medium about 30% smaller (in medium,
for some kinds of data the size is the same as for gzip).
So no, there's no reason against it, just some time is need to adapt
everything.
--
IOnut - Un^d^dregistered ;) FreeBSD user
Intellectual Property is nowhere near as valuable as Intellect
FreeBSD committer - ite...@freebsd.org, PGP Key ID 057E9F8B493A297B
signature.asc
Description: PGP signature
