Re: LZO support
Okay, I have become a little more advanced lately, and was able to compile the kernel zlib and lzo in userspace (last year I gave up...). So here is the code (with results, the lib file is the compiled lzo 2.03): http://wiki.laptop.org/images/5/5b/Zlibtest5.tar.gz Since I did not find any assembly compressor support (only decompressor), I do not know what did I talked about last time. It could have been that I have been dreaming about that... The program needs a test.dat file, I have used cat */*test.dat in /var/lib which gave me a 22 MB file (the code handles a max of 40 MB). The results are consistent with Erik's results. Since ZLIB has a compression level parameter, and LZO has different compressor types, I will test those as well. Now I think that jffs2 (or any other flash filesystem) should use LZO to store the data, so if the limit is the flash write speed then it could write twice as fast. Later, when there is ample power in the battery and no cpu utilization, it should recompress the data either using a better LZO or by ZLIB (of course only those files which are old enough, so it would not recompress files which will be modified soon). And the results: started with block size == 4096 LZO test in size: 22987994 22987994 - 11575529 (50.354672%) 1.96 seconds - 11.185234 MB/sec 11575529 - 22987994 0.75 seconds - 29.230745 MB/sec compared 22987994 bytes OK LZO asm test in size: 22987994 22987994 - 11575529 (50.354672%) 1.93 seconds - 11.359098 MB/sec 11575529 - 22987994 0.49 seconds - 44.740936 MB/sec compared 22987994 bytes OK ZLIB test in size: 22987994 22987994 - 8982973 (39.076802%) 9.81 seconds - 2.234767 MB/sec 8982973 - 22987994 2.75 seconds - 7.972022 MB/sec compared 22987994 bytes OK Greg Smith wrote: Hi Erik, Can you design a test case or two to test the performance of these compression schemes? Thanks, Greg S ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
How are you producing the test data (test.dat) used by the test? On Sun, Jul 20, 2008 at 10:57:37AM +0200, NoiseEHC wrote: Okay, I have become a little more advanced lately, and was able to compile the kernel zlib and lzo in userspace (last year I gave up...). So here is the code (with results, the lib file is the compiled lzo 2.03): http://wiki.laptop.org/images/5/5b/Zlibtest5.tar.gz Since I did not find any assembly compressor support (only decompressor), I do not know what did I talked about last time. It could have been that I have been dreaming about that... The program needs a test.dat file, I have used cat */*test.dat in /var/lib which gave me a 22 MB file (the code handles a max of 40 MB). The results are consistent with Erik's results. Since ZLIB has a compression level parameter, and LZO has different compressor types, I will test those as well. Now I think that jffs2 (or any other flash filesystem) should use LZO to store the data, so if the limit is the flash write speed then it could write twice as fast. Later, when there is ample power in the battery and no cpu utilization, it should recompress the data either using a better LZO or by ZLIB (of course only those files which are old enough, so it would not recompress files which will be modified soon). And the results: started with block size == 4096 LZO test in size: 22987994 22987994 - 11575529 (50.354672%) 1.96 seconds - 11.185234 MB/sec 11575529 - 22987994 0.75 seconds - 29.230745 MB/sec compared 22987994 bytes OK LZO asm test in size: 22987994 22987994 - 11575529 (50.354672%) 1.93 seconds - 11.359098 MB/sec 11575529 - 22987994 0.49 seconds - 44.740936 MB/sec compared 22987994 bytes OK ZLIB test in size: 22987994 22987994 - 8982973 (39.076802%) 9.81 seconds - 2.234767 MB/sec 8982973 - 22987994 2.75 seconds - 7.972022 MB/sec compared 22987994 bytes OK Greg Smith wrote: Hi Erik, Can you design a test case or two to test the performance of these compression schemes? Thanks, Greg S ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Sun, Jul 20, 2008 at 04:38:36PM -0400, Erik Garrison wrote: How are you producing the test data (test.dat) used by the test? Excuse me. You clearly stated that in your email! On Sun, Jul 20, 2008 at 10:57:37AM +0200, NoiseEHC wrote: The program needs a test.dat file, I have used cat */*test.dat in /var/lib which gave me a 22 MB file (the code handles a max of 40 MB). The results are consistent with Erik's results. Since ZLIB has a compression level parameter, and LZO has different compressor types, I will test those as well. ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
Hi Erik, Can you design a test case or two to test the performance of these compression schemes? Thanks, Greg S ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Tue, Jul 15, 2008 at 04:41:35AM -0400, Greg Smith wrote: Hi Erik, Can you design a test case or two to test the performance of these compression schemes? Yes. The following script encodes a trivial and necessarily flawed test case: http://dev.laptop.org/~erik/code/testcompression.py To run this test script you will need http://www.zlib.net/zpipe.c, a simple zlib utilization which compresses or decompresses stdin to stdout. (compile with gcc zpipe.c -o zpipe -lz) somewhere in your path. Additionally, you will need to install lzop there are packages on both ubuntu and fedora (yum install lzop; sudo aptitude install lzop). The script runs compression and decompression of a target file (both to /dev/null to mitigate I/O bounding) some number of times, averages the results, and then prints the ratios between each algorithm's compression ratio, compression time, and decompression time. The idea is to get a rough estimate of the potential performance benefits of one algorithm over the other by utilizing them both in the same environment. % python testcompression.py --help Usage: testcompression.py [options] Options: -h, --helpshow this help message and exit -c CYCLES, --cycles=CYCLES run each compression or decompression CYCLES times A simple test (on the uncompressed joyride usb image os2163.usb) is as follows: % wget http://xs-dev.laptop.org/~cscott/xo-1/streams/joyride/build2163/devel_jffs2/os2163.usb % python testcompression.py -c 5 os2163.usb running 5 cycles testing on os2163.usb size = 706715648 writing lzo compressed file os2163.usb.lzo writing zlib compressed file os2163.usb.zlib lzo compression . lzo decompression . zlib compression . zlib decompression . results on os2163.usb lzo: compression ratio 0.511157, compress 9.281282 s, decompress 2.848002 s zlib: compression ratio 0.401923, compress 51.469934 s, decompress 4.607102 s ratios, lzo : zlib --- compression ratio: 1.2715711 --- compression time: 0.180324344864 --- decompression time: 0.618176592161 These results (lzo takes 1/5th the time to compress, 1/2 the time to decompress, and takes up 120% of the space as zlib) seem to be roughly in line with my other tests. Possible flaws: - the algorithms use a different blocksize than our fs - the algorithms are significantly different from those included in the kernel I would be happy to hear more flaws or ideas about how to improve this test. Erik ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
It seems that you two did not notice the smiley at the end of that sentence... I rather not to think about that it was not funny... :) First, I have to apologize since that statement was made by Mitch Bradley here: http://lists.laptop.org/pipermail/devel/2008-January/009437.html I am sorry, seems that I am just too old to remember names anymore. For excuse I could say that both names starts with M so it was close... :) Now about my obsession of compression code tuning: Lately I have more or less finished documenting the Geode processor and need some problem where I can test what I have learned. Since compression is small, has potential and relatively isolated, I have chosen that. It could have been anything else but it seems that this can have the most effect. C. Scott Ananian wrote: On Mon, Jul 14, 2008 at 6:01 PM, Michael Stone [EMAIL PROTECTED] wrote: On Mon, Jul 14, 2008 at 11:51:20PM +0200, Tomeu Vizoso wrote: On Mon, Jul 14, 2008 at 9:00 PM, NoiseEHC [EMAIL PROTECTED] wrote: I have a partially finished ZLIB decompression code as well. mstone just told me that time that we will use LZO so that effort was moot... Currently I cannot decide if I should cry or just laugh hysterically... :) I believe I made statements a few months ago about what compression we presently _do_ use, but I don't think I commented on what compression we ought to be using. Can you remind me of my words? And, for the record, I didn't say we'll be using zlib instead -- in reality, we are trying to move off jffs2 just as quickly as possible, and the compression scheme we use next will be heavily influenced by whatever the other filesystem has adopted. But it is a fact that we are currently using zlib and rtime compression in jffs2, and OFW doesn't currently support lzo compression, so that adoption of LZO in jffs2 would have to wait until partition support -- which is also roughly the time when I hope to replace the non-boot partitions with something better than jffs2. So the window for LZO-related jffs2 improvements is very narrow. If logfs/ubifs/yaffs can benefit from your LZO work, then you've multiplied its potential usefulness -- and a quick google shows discussions of LZO in ubifs. Besides, improving LZO in the kernel improves things for everyone, not just OLPC, so if it's your itch, go scratch it! --scott ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Tue, Jul 15, 2008 at 07:06:53PM +0200, NoiseEHC wrote: Also for speed testing this script seems totally unusable, I would send my c program but it does not contain ZLIB. Do you mean that running the tests inside a python script is bad for speed testing? No, the problem is that you do not know what amount of processing is happening because of reading and piping files, or if the compressors/decompressors are not written in a brain damaged way, or if the extra processing trashes caches and so on. If you think that zpipe/lzop are fine then it is usable as a speed test. You also has to use the minimum time, not the average, if you have multiple runs. I understand. For the amount of time I can give to this I'm going to accept that zpipe/lzop are fine. The one major change which I can implement is taking the minimum decompression time instead of the average. I have updated the test script to do so. I'm not observing a significant change in the results. Also there is a problem that the LZO included in the kernel is a modified version of minilzo (x1 LZO algorithm) and the default compressor does not generate all the possible codes during compression. Where could I read more about this? Nowhere. Currently the only documentation of the LZO file format is written by me (attached). There is a file lzo1.c in the original LZO library which describes a hypothetical, non existing format, unfortunately the 1x_1 format which is in minilzo.c which was kernelized is undocumented spaghetti code... Could you venture a guess at how much time would be required to solve this, or at least how large the changes to the kernel's codebase would have to be? These ones are not used: ; short match???, can only happen after a normal literal mmll match op-0x801-m,3 ; literal ll ; match type = 15 (m 0x400) (op-0x400..op-1) mmll match op-1-m,2 ; literal ll ...,...0 ...,...1 ...,...2 ...,...3 ; normal literal, not followed with a short literal (literal ll) literal t+3 (3..18) + literal 15+T+3 (19..) ; short match???, can only happen after a normal literal mmll match op-0x801-m,3 ; literal ll ; match type = 64 (t = 2) (m 0x800) (op-0x800..op-1) tttmmmll match op-1-m,t-1+2 ; literal ll ; match type = 32 (m 0x4000) (op-0x4000..op-1) 001t mmll match op-1-m,t+2 (3..33) ; literal ll 0010 + mmll match op-1-m,31+T+2 (34..) ; literal ll ; match type = 16 (m 0x8000) (op-0xbfff..op-0x4001) 00010xxx 00yy eof (where xxx != 000) 0001mttt mmll match op-m-0x4000,t+2 (3..9) (where m != 0) ; literal ll 0001m000 + mmll match op-m-0x4000,7+T+2 (10..) (where m != 0) ; literal ll ; match type = 15 (m 0x400) (op-0x400..op-1) mmll match op-1-m,2 ; literal ll ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Tue, Jul 15, 2008 at 01:05:42PM -0400, Erik Garrison wrote: On Tue, Jul 15, 2008 at 04:41:35AM -0400, Greg Smith wrote: Hi Erik, Can you design a test case or two to test the performance of these compression schemes? Yes. The following script encodes a trivial and necessarily flawed test case: http://dev.laptop.org/~erik/code/testcompression.py Per suggestions by NoiseEHC I have updated the test script to base its report on the minimum times for compression and decompression. The results do not appear to be significantly different: % python testcompression.py -c 10 os2163.usb running 10 cycles testing on os2163.usb size = 706715648 writing lzo compressed file os2163.usb.lzo writing zlib compressed file os2163.usb.zlib lzo compression .. lzo decompression .. zlib compression .. zlib decompression .. results on os2163.usb using minimum runtimes lzo: compression ratio 0.511157, compress 8.925809 s, decompress 2.632027 s zlib: compression ratio 0.401923, compress 50.913132 s, decompress 4.799342 s ratios, lzo : zlib --- compression ratio: 1.2715711 --- compression time: 0.175314471382 --- decompression time: 0.54841408952 Erik ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
Hi, Based on some very provisional testing, I believe that the extremely long save times we're seeing in EToys are at least partly related to our use of zlib compression for data nodes in the filesystem. We could switch to LZO compression for a 500% speedup in compression (write), a 50% speedup in decompression (read), and a slight (but significant) decrease in compression ratio. I prefer aiming at #2886 (Some files shouldn't be compressed by jffs2), which goes a long way to solving the user problems without introducing the logistical cost of LZO support in OFW or partitions. Saving a movie would be faster still than LZO if we could just tell jffs2 that we're giving it compressed data that it shouldn't recompress. - Chris. -- Chris Ball [EMAIL PROTECTED] ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Erik Garrison wrote: | But an alternative is to implement a | partitioning scheme in which user data resides on a LZO-compressed | partition and the system data resides on a zlib-compressed partition. I strongly favor this approach. Regardless of compression method, JFFS2 was not designed for 1 GB flash. In fact, JFFS2 was chosen when the XO's disk was only 512 MB. With the coming red XO upgrade to larger flash sizes, we will need to replace JFFS2. Reimplementing every filesystem in OFW is infeasible, especially while we are testing a variety of new filesystems. Even if we remain with JFFS2, partitioning promises to reduce our boot time substantially on the current hardware. The support is largely in place. Someone just needs to make it happen. - --Ben -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkh7fkkACgkQUJT6e6HFtqT8pgCfRkCfLdp8+dwTgS2jFjLOo42N XdsAniAz9iIOaiGMj8+qNXa4g69F1uaa =c+yo -END PGP SIGNATURE- ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Mon, Jul 14, 2008 at 6:19 PM, Chris Ball [EMAIL PROTECTED] wrote: Hi, Based on some very provisional testing, I believe that the extremely long save times we're seeing in EToys are at least partly related to our use of zlib compression for data nodes in the filesystem. We could switch to LZO compression for a 500% speedup in compression (write), a 50% speedup in decompression (read), and a slight (but significant) decrease in compression ratio. I prefer aiming at #2886 (Some files shouldn't be compressed by jffs2), which goes a long way to solving the user problems without introducing the logistical cost of LZO support in OFW or partitions. +100 Most data that gets into the journal is already compressed: PNG, OGG, PDF (some), ODT, XO, XOL, etc. How would this work? Should we create an empty file, set an xattr and then start writing? If so, we should think about how we can modify all sw to do so. Alternatively, as in most cases data is already compressed, perhaps would be better to not compress by default - or let that process to zip the file with gzip :P. If jffs2 can check if the first block compresses and stop doing it if it doesn't, that may be best. Regards, Tomeu ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support (was: [Re: low power actions?])
On Jul 14 2008, at 12:07, Erik Garrison was caught saying: There is an existing ticket to address this issue (http://dev.laptop.org/ticket/2072). This ticket suggests that we could add LZO support to OFW in order to implement LZO-compressed data nodes in the filesystem. In this scheme OFW would require LZO support to boot a kernel which resides on an LZO-compressed fs. But an alternative is to implement a partitioning scheme in which user data resides on a LZO-compressed partition and the system data resides on a zlib-compressed partition. This would provide us with good compression for system components (saving space) but would also allow users to write data much more quickly. That said, implementing LZO compression across the entire system might improve system responsiveness by a noticeable margin, so it might be worthwhile. Testing is required. Partitioning is already on our roadmap, so if we just put /boot and other OFW needed data in a non-LZO partition, we can live w/o needing to update OFW to handle LZO. ~Deepak -- Deepak Saxena [EMAIL PROTECTED] ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
Hi, I prefer aiming at #2886 (Some files shouldn't be compressed by jffs2), which goes a long way to solving the user problems without introducing the logistical cost of LZO support in OFW or partitions. How would this work? Should we create an empty file, set an xattr and then start writing? If so, we should think about how we can modify all sw to do so. Alternatively, as in most cases data is already compressed, perhaps would be better to not compress by default - or let that process to zip the file with gzip :P. Deepak suggests (in #2886) using something like an O_NOCOMPRESSION flag to open(2), which sounds fine to me. We'd either pass it to Python's open() (how?) or set it immediately after open with fcntl(, F_SETFL, ). - Chris. -- Chris Ball [EMAIL PROTECTED] ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Jul 14 2008, at 12:19, Chris Ball was caught saying: Hi, Based on some very provisional testing, I believe that the extremely long save times we're seeing in EToys are at least partly related to our use of zlib compression for data nodes in the filesystem. We could switch to LZO compression for a 500% speedup in compression (write), a 50% speedup in decompression (read), and a slight (but significant) decrease in compression ratio. I prefer aiming at #2886 (Some files shouldn't be compressed by jffs2), which goes a long way to solving the user problems without introducing the logistical cost of LZO support in OFW or partitions. I've been thinking about this a bit and there are all sorts of issues involved here. The patches I know of that that exist to disable compression on cramfs are for use at file system creation time, where we know exactly what data we're stuffing into the system. These are primarilly used to disable compression on executables and allow XIP. If we want to do this at runtime, we have to figure out how to tell the the kernel not to compress a file? - One option, as suggested in the trac, is to have the kernel trap file extensions and disable compression based on that. Doable, but I expect a rather large hack to VFS to do this. An extension to this idea is to look for specific signatures in files that disable compression. - Another option is to provide a one-time flag passed at file creation time (O_CREAT) to enable/disable compression. This would also require VFS changes but I think would be simpler; however, this case has the problem that all applications that may at one time write a file that does not need compression need to be modified to support this extension. Saving a movie would be faster still than LZO if we could just tell jffs2 that we're giving it compressed data that it shouldn't recompress. Do we have some idea of how much of our data in general is compressed and how much is not? If most our compressed data is small text files and the majority or our data is large compressed files, maybe we can drop the compression requirement all together? If we look beyond jffs2 and consider yaffs as an alternative (not upstream but has been actively used in the embedded world for years), it does no compression whatsoever at the filesystem level. ~Deepak -- Deepak Saxena [EMAIL PROTECTED] ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Mon, Jul 14, 2008 at 12:50:26PM -0400, Chris Ball wrote: Hi, I prefer aiming at #2886 (Some files shouldn't be compressed by jffs2), which goes a long way to solving the user problems without introducing the logistical cost of LZO support in OFW or partitions. How would this work? Should we create an empty file, set an xattr and then start writing? If so, we should think about how we can modify all sw to do so. Alternatively, as in most cases data is already compressed, perhaps would be better to not compress by default - or let that process to zip the file with gzip :P. Deepak suggests (in #2886) using something like an O_NOCOMPRESSION flag to open(2), which sounds fine to me. We'd either pass it to Python's open() (how?) or set it immediately after open with fcntl(, F_SETFL, ). This approach will work provided all data is saved through the journal. Note that in at least one application (Record) which must write directly to the fs (afaik this is because of gstreamer). Is this the only application with such a requirement? If it is, are we expecting it to remain the only such application? In my opinion we should assume that other applications will have similar requirements, and work to implement compression at a lower level in the software stack so that we don't need to manage it at this level. That said, I personally support any small hack which we can push to the Journal to quickly implement a stopgap solution. Erik ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Mon, Jul 14, 2008 at 7:07 PM, Erik Garrison [EMAIL PROTECTED] wrote: On Mon, Jul 14, 2008 at 12:50:26PM -0400, Chris Ball wrote: Hi, I prefer aiming at #2886 (Some files shouldn't be compressed by jffs2), which goes a long way to solving the user problems without introducing the logistical cost of LZO support in OFW or partitions. How would this work? Should we create an empty file, set an xattr and then start writing? If so, we should think about how we can modify all sw to do so. Alternatively, as in most cases data is already compressed, perhaps would be better to not compress by default - or let that process to zip the file with gzip :P. Deepak suggests (in #2886) using something like an O_NOCOMPRESSION flag to open(2), which sounds fine to me. We'd either pass it to Python's open() (how?) or set it immediately after open with fcntl(, F_SETFL, ). This approach will work provided all data is saved through the journal. Note that in at least one application (Record) which must write directly to the fs (afaik this is because of gstreamer). Is this the only application with such a requirement? If it is, are we expecting it to remain the only such application? In my opinion we should assume that other applications will have similar requirements, and work to implement compression at a lower level in the software stack so that we don't need to manage it at this level. That said, I personally support any small hack which we can push to the Journal to quickly implement a stopgap solution. In the current DS API, activities write the file to the main fs, and then ask the DS to move (preferred) or copy that file into the DS space. So we would need to modify all activities to set that flag after opening the file and before writing to it. Regards, Tomeu ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
Hi, - One option, as suggested in the trac, is to have the kernel trap file extensions and disable compression based on that. Doable, but I expect a rather large hack to VFS to do this. An extension to this idea is to look for specific signatures in files that disable compression. Not sure we need to go this far. - Another option is to provide a one-time flag passed at file creation time (O_CREAT) to enable/disable compression. This would also require VFS changes but I think would be simpler; however, this case has the problem that all applications that may at one time write a file that does not need compression need to be modified to support this extension. I think this would get us 90% of the way there with a little work. I don't care about arbitrary code being able to set no-compression as much as I care about code we control being able to do so. Do we have some idea of how much of our data in general is compressed and how much is not? Yes: -rw-r--r-- 1 cjb cjb 928M 2008-07-14 01:57 joyride-2158.ext3.img -rw-r--r-- 1 cjb cjb 349M 2008-07-14 01:57 joyride-2158.jffs2.img -rw-r--r-- 1 cjb cjb 264M 2008-07-14 01:57 joyride-2158.ext3.img.bz2 - Chris. -- Chris Ball [EMAIL PROTECTED] ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Mon, Jul 14, 2008 at 1:27 PM, Chris Ball [EMAIL PROTECTED] wrote: - Another option is to provide a one-time flag passed at file creation time (O_CREAT) to enable/disable compression. This would also require VFS changes but I think would be simpler; however, this case has the problem that all applications that may at one time write a file that does not need compression need to be modified to support this extension. I think this would get us 90% of the way there with a little work. I don't care about arbitrary code being able to set no-compression as much as I care about code we control being able to do so. I prefer an xattr being set in addition, so that the 'no compression' information is saved across backup/restore etc. I'd prefer in place of instead of in addition to, but the POSIX apis for xattrs would make this really inconvenient on the kernel side: you'd have to open the file, then set the xattr, *then* write the data. Do we have some idea of how much of our data in general is compressed and how much is not? Yes: -rw-r--r-- 1 cjb cjb 928M 2008-07-14 01:57 joyride-2158.ext3.img -rw-r--r-- 1 cjb cjb 349M 2008-07-14 01:57 joyride-2158.jffs2.img -rw-r--r-- 1 cjb cjb 264M 2008-07-14 01:57 joyride-2158.ext3.img.bz2 Look also at the .usb file, which is an uncompressed ISO9660 image of our filesystem. Using an uncompressed filesystem is really not an option. --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Mon, Jul 14, 2008 at 7:27 PM, Chris Ball [EMAIL PROTECTED] wrote: - Another option is to provide a one-time flag passed at file creation time (O_CREAT) to enable/disable compression. This would also require VFS changes but I think would be simpler; however, this case has the problem that all applications that may at one time write a file that does not need compression need to be modified to support this extension. I think this would get us 90% of the way there with a little work. I don't care about arbitrary code being able to set no-compression as much as I care about code we control being able to do so. What about not compressing by default? Then modify pilgrim to compress depending on the kind of file (or compress all of them). And then let activities that want to compress their data to do so? I can only think of etoys as needing it, any other? Regards, Tomeu ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Mon, Jul 14, 2008 at 1:34 PM, Tomeu Vizoso [EMAIL PROTECTED] wrote: On Mon, Jul 14, 2008 at 7:27 PM, Chris Ball [EMAIL PROTECTED] wrote: - Another option is to provide a one-time flag passed at file creation time (O_CREAT) to enable/disable compression. This would also require VFS changes but I think would be simpler; however, this case has the problem that all applications that may at one time write a file that does not need compression need to be modified to support this extension. I think this would get us 90% of the way there with a little work. I don't care about arbitrary code being able to set no-compression as much as I care about code we control being able to do so. What about not compressing by default? Then modify pilgrim to compress depending on the kind of file (or compress all of them). And then let activities that want to compress their data to do so? I can only think of etoys as needing it, any other? That might be reasonable to try. mkjffs2 will always try zlib and rtime compression when it creates the filesystem; all that would be required is to build a kernel where jffs2 did not try to compress by default (which is different than disabling compression support). I guess the journal could always take over the burden of compressing its entries if needed. --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
Currently I am profiling/rewriting the LZO decompression code. The speed result (from memory, not too reliable): LZO compression: ~9.5 MB/sec LZO compression ASM version: ~9.3 MB/sec LZO decompression: ~25MB/sec LZO decompression ASM version: ~40 MB/sec Here the ASM version is the version from the original LZO library. The test were made with a 40MB file which was created cat-ing data from /lib (as I remember...), it is half .so and half text, compression is to 50-60%. See, the ASM optimized version is slower than the kernel's one (lzo1x_compress.c). I am shooting at ~80-100 MB/sec, it would make it worthwhile to include in the kernel. Note that if this succeeds then loading will become IO limited instead of CPU limited. Could somebody make an LZO compressed image? Unfortunately I am not qualified for this work. ps: This work stalled a little bit since it seems the people who converted minilzo to be compilable with the kernel messed up the code. Erik Garrison wrote: On Fri, Jul 11, 2008 at 06:33:41PM +0100, Gary C Martin wrote: One concern I have with auto saving state before powering off is the potential corruption of journal data. How robust is the Journal if power off happens half way through an ongoing auto state save – do you loose both the new journal entry and the original entry you had resumed from (partially overwritten)? The worst case just now seems to be EToys, it's really great for tinkering, but creating even simple scripts seems to require a massive amount of time to save (example 5+min!! to close save a script with five polygon space invaders with 2 frames of animation and a few lines of movement code). Add to that that EToys seems to be almost as power intensive as video recording (my impression only, I've not power tested). I find I'm the habit of plugging into mains before I try and close an EToys instance I've made more than a few modifications to. Based on some very provisional testing, I believe that the extremely long save times we're seeing in EToys are at least partly related to our use of zlib compression for data nodes in the filesystem. We could switch to LZO compression for a 500% speedup in compression (write), a 50% speedup in decompression (read), and a slight (but significant) decrease in compression ratio. While working with Record I noticed that we cannot write very high resolution video (320x240 @ 10fps) to the filesystem because of this issue. It would help to know what other applications are suffering because of poor fs write speed. There is an existing ticket to address this issue (http://dev.laptop.org/ticket/2072). This ticket suggests that we could add LZO support to OFW in order to implement LZO-compressed data nodes in the filesystem. In this scheme OFW would require LZO support to boot a kernel which resides on an LZO-compressed fs. But an alternative is to implement a partitioning scheme in which user data resides on a LZO-compressed partition and the system data resides on a zlib-compressed partition. This would provide us with good compression for system components (saving space) but would also allow users to write data much more quickly. That said, implementing LZO compression across the entire system might improve system responsiveness by a noticeable margin, so it might be worthwhile. Testing is required. Erik ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Mon, Jul 14, 2008 at 2:05 PM, NoiseEHC [EMAIL PROTECTED] wrote: Currently I am profiling/rewriting the LZO decompression code. Again, be warned that there's no guarantee we will ever use LZO compression. The plan of record is to disable compression on selected files instead. That's not to say you shouldn't do the work if it interests you, just that you'll be facing an uphill battle don't say we didn't warn you. =) worthwhile to include in the kernel. Note that if this succeeds then loading will become IO limited instead of CPU limited. Are your benchmarks catting data from a jffs2 partition on FLASH or an ext2/3 partition on a hard drive? I'd be interested to know exactly how far we are from being IO limited at present. Are we quibbling about details in the noise? --scott ps. even with the above warning, there are many scenarios we might find ourselves using LZO on jffs2, at least for a short while, so good luck! -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
C. Scott Ananian wrote: On Mon, Jul 14, 2008 at 2:05 PM, NoiseEHC [EMAIL PROTECTED] wrote: Currently I am profiling/rewriting the LZO decompression code. Again, be warned that there's no guarantee we will ever use LZO compression. The plan of record is to disable compression on selected files instead. That's not to say you shouldn't do the work if it interests you, just that you'll be facing an uphill battle don't say we didn't warn you. =) I have a partially finished ZLIB decompression code as well. mstone just told me that time that we will use LZO so that effort was moot... Currently I cannot decide if I should cry or just laugh hysterically... :) worthwhile to include in the kernel. Note that if this succeeds then loading will become IO limited instead of CPU limited. Are your benchmarks catting data from a jffs2 partition on FLASH or an ext2/3 partition on a hard drive? I'd be interested to know exactly how far we are from being IO limited at present. Are we quibbling about details in the noise? --scott These benchmarks just measure the noncached memory - memory compression. As I have said I am not too qualified for this work, so unless somebody creates an XO image which uses LZO... I mean that if there is a working F9 image with the new kernel then I will be able to work from the srpm but simply cannot build a full image myself. ps. even with the above warning, there are many scenarios we might find ourselves using LZO on jffs2, at least for a short while, so good luck! ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Mon, Jul 14, 2008 at 11:51:20PM +0200, Tomeu Vizoso wrote: On Mon, Jul 14, 2008 at 9:00 PM, NoiseEHC [EMAIL PROTECTED] wrote: I have a partially finished ZLIB decompression code as well. mstone just told me that time that we will use LZO so that effort was moot... Currently I cannot decide if I should cry or just laugh hysterically... :) I believe I made statements a few months ago about what compression we presently _do_ use, but I don't think I commented on what compression we ought to be using. Can you remind me of my words? Regards, Michael ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: LZO support
On Mon, Jul 14, 2008 at 6:01 PM, Michael Stone [EMAIL PROTECTED] wrote: On Mon, Jul 14, 2008 at 11:51:20PM +0200, Tomeu Vizoso wrote: On Mon, Jul 14, 2008 at 9:00 PM, NoiseEHC [EMAIL PROTECTED] wrote: I have a partially finished ZLIB decompression code as well. mstone just told me that time that we will use LZO so that effort was moot... Currently I cannot decide if I should cry or just laugh hysterically... :) I believe I made statements a few months ago about what compression we presently _do_ use, but I don't think I commented on what compression we ought to be using. Can you remind me of my words? And, for the record, I didn't say we'll be using zlib instead -- in reality, we are trying to move off jffs2 just as quickly as possible, and the compression scheme we use next will be heavily influenced by whatever the other filesystem has adopted. But it is a fact that we are currently using zlib and rtime compression in jffs2, and OFW doesn't currently support lzo compression, so that adoption of LZO in jffs2 would have to wait until partition support -- which is also roughly the time when I hope to replace the non-boot partitions with something better than jffs2. So the window for LZO-related jffs2 improvements is very narrow. If logfs/ubifs/yaffs can benefit from your LZO work, then you've multiplied its potential usefulness -- and a quick google shows discussions of LZO in ubifs. Besides, improving LZO in the kernel improves things for everyone, not just OLPC, so if it's your itch, go scratch it! --scott -- ( http://cscott.net/ ) ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
