Re: LZMA inclusion

2008-12-16 Thread Lasse Collin
Geert Uytterhoeven wrote: On Tue, 9 Dec 2008, Lasse Collin wrote: Is it OK that the decoder allocates memory in the middle of decoding, or must all the memory be preallocated when the decoder is initialized? If everything has to be preallocated, then the initialization function needs an

Re: LZMA inclusion

2008-12-09 Thread Geert Uytterhoeven
On Tue, 9 Dec 2008, Lasse Collin wrote: Phillip Lougher wrote: The issue that moving to 4 KiB output buffers solves is it reduces significantly the number of 1 MiB (or 128 KiB for the default block size) buffers that need to be vmalloced. Squashfs caches the last 3 fragment buffers read,

Re: LZMA inclusion

2008-12-09 Thread Lasse Collin
Phillip Lougher wrote: The issue that moving to 4 KiB output buffers solves is it reduces significantly the number of 1 MiB (or 128 KiB for the default block size) buffers that need to be vmalloced. Squashfs caches the last 3 fragment buffers read, and moving to 4 KiB buffers eliminates these

Re: LZMA inclusion

2008-12-08 Thread Jamie Lokier
Phillip Lougher wrote: One-shot LZMA decoding therefore isn't going to work very well with future versions of Squashfs, obviously a solution (as is currently done with the Squashfs-LZMA patches) is to use separately allocated contiguous input/output buffers, and memcpy into and out of them,

Re: LZMA inclusion

2008-12-08 Thread Lasse Collin
Phillip Lougher wrote: Currently, as mentioned above, Squashfs decompresses into a single contiguous output buffer. But, due to the linux kernel mailing list's dislike of vmalloc, this is being changed. In future Squashfs will decompress into a sequence of 4 KiB output buffers (possibly in

Re: LZMA inclusion

2008-12-08 Thread Phillip Lougher
Lasse Collin wrote: Phillip Lougher wrote: Currently, as mentioned above, Squashfs decompresses into a single contiguous output buffer. But, due to the linux kernel mailing list's dislike of vmalloc, this is being changed. In future Squashfs will decompress into a sequence of 4 KiB output

Re: LZMA inclusion

2008-12-08 Thread Jörn Engel
On Sun, 7 December 2008 23:32:32 +, Phillip Lougher wrote: Currently, as mentioned above, Squashfs decompresses into a single contiguous output buffer. But, due to the linux kernel mailing list's dislike of vmalloc, this is being changed. Don't blame lkml, blame Intel and IBM. Back

Re: LZMA inclusion

2008-12-08 Thread Phillip Lougher
Jörn Engel wrote: On Sun, 7 December 2008 23:32:32 +, Phillip Lougher wrote: Currently, as mentioned above, Squashfs decompresses into a single contiguous output buffer. But, due to the linux kernel mailing list's dislike of vmalloc, this is being changed. Don't blame lkml, blame Intel

Re: LZMA inclusion

2008-12-08 Thread Jörn Engel
On Mon, 8 December 2008 21:47:37 +, Phillip Lougher wrote: Yes, I'm aware of the issues with vmalloc on older hardware. It's not even limited to older hardware. Blue Gene supercomputers are large clusters of ppc440 machines. Iirc each node consists of two 32bit cpus and up to 4GB of RAM.

Re: LZMA inclusion

2008-12-07 Thread Phillip Lougher
Jörn Engel wrote: On Sat, 6 December 2008 23:56:50 +0200, Lasse Collin wrote: Since you are improving the crypto API, maybe it would be a good idea to add a flag to tell the decoder that the whole output buffer will be kept available to the multi-call decoder. I'm not convinced this is the

Re: LZMA inclusion

2008-12-06 Thread Lasse Collin
Geert Uytterhoeven wrote: SquashFS does not do one-shot decompression, it calls the decompressor multiple times, after reading each block. I guess that can be changed, but it will increase memory pressure, as blocks cannot be released until the decompression has finished. I read the code more

Re: LZMA inclusion

2008-12-05 Thread Geert Uytterhoeven
On Wed, 3 Dec 2008, Lasse Collin wrote: Once the stable release of the xz package is out, I could be willing to write an easy-to-use .xz decoder that is suitable for inclusion to Linux (including proper coding style with comments). I have understood, that for kernel and initramfs

Re: LZMA inclusion

2008-12-04 Thread Jean-Christophe PLAGNIOL-VILLARD
The primary compression algorithm of the .xz format is LZMA2. It fixes some practical problems of the original LZMA. The .xz format also supports filters for executable data (x86, ARM and a few others), which combined with LZMA2 usually improve compression ratio 5-10 % over plain LZMA or

Re: LZMA inclusion

2008-12-03 Thread Tim Bird
Gregers Petersen wrote: There was a small talk a few days ago involving a few of the OpenWrt developers and David Woodhouse. One of the topics discussed, was a question about the potential of including LZMA in the kernel. Such an inclusion would be quite benefitial in terms of embedded

Re: LZMA inclusion

2008-12-03 Thread Florian Fainelli
Hi Gregers, Tim, Le Wednesday 03 December 2008 20:36:45 Tim Bird, vous avez écrit : Did anyone answer this? CELF is currently considering funding a project to do this (add LZMA support to the kernel), and it would be good to get a feel for the current status... There has been an effort for

Re: LZMA inclusion

2008-12-03 Thread Bernhard Reutner-Fischer
On Wed, Dec 03, 2008 at 11:36:45AM -0800, Tim Bird wrote: Gregers Petersen wrote: There was a small talk a few days ago involving a few of the OpenWrt developers and David Woodhouse. One of the topics discussed, was a question about the potential of including LZMA in the kernel. Such an

Re: LZMA inclusion

2008-12-03 Thread Gregers Petersen
Tim Bird wrote: Gregers Petersen wrote: There was a small talk a few days ago involving a few of the OpenWrt developers and David Woodhouse. One of the topics discussed, was a question about the potential of including LZMA in the kernel. Such an inclusion would be quite benefitial in terms of

Re: LZMA inclusion

2008-12-03 Thread Sam Ravnborg
On Wed, Dec 03, 2008 at 08:58:52PM +0100, Bernhard Reutner-Fischer wrote: On Wed, Dec 03, 2008 at 11:36:45AM -0800, Tim Bird wrote: Gregers Petersen wrote: There was a small talk a few days ago involving a few of the OpenWrt developers and David Woodhouse. One of the topics discussed, was a

Re: LZMA inclusion

2008-12-03 Thread Bernhard Reutner-Fischer
On Wed, Dec 03, 2008 at 09:20:09PM +0100, Sam Ravnborg wrote: On Wed, Dec 03, 2008 at 08:58:52PM +0100, Bernhard Reutner-Fischer wrote: On Wed, Dec 03, 2008 at 11:36:45AM -0800, Tim Bird wrote: Gregers Petersen wrote: There was a small talk a few days ago involving a few of the OpenWrt

Re: LZMA inclusion

2008-12-03 Thread Sam Ravnborg
PS: Not sure if you, Sam, are the right person who cares for it, but i think that the help-text and actual accepted arguments of scripts/kconfig/lxdialog/check-lxdialog.sh are out of sync. I queued up the following: From f6682f915760ccfe57ef1b6cd5ff2d8f2bf8c1d4 Mon Sep 17 00:00:00 2001 From:

Re: LZMA inclusion

2008-12-03 Thread Bernhard Reutner-Fischer
On Wed, Dec 03, 2008 at 10:16:17PM +0100, Sam Ravnborg wrote: PS: Not sure if you, Sam, are the right person who cares for it, but i think that the help-text and actual accepted arguments of scripts/kconfig/lxdialog/check-lxdialog.sh are out of sync. I queued up the following: Thanks alot.

Re: LZMA inclusion

2008-12-03 Thread Sam Ravnborg
two into a simple make ... archprepare headers_install: Which architectures needs this archprepare? At least cris tripped this IIRC, at least before or around the time when cris's subarch handling was fixed (the asm-cris/arch-v10 vs. arch-v32 linking issue). It would be good to get rid

LZMA inclusion

2008-11-24 Thread Gregers Petersen
Hello There was a small talk a few days ago involving a few of the OpenWrt developers and David Woodhouse. One of the topics discussed, was a question about the potential of including LZMA in the kernel. Such an inclusion would be quite benefitial in terms of embedded systems, but the major