On Wed, Jan 16, 2002 at 02:05:10PM -0500, [EMAIL PROTECTED] wrote: > On Wed, 16 Jan 2002 15:22:45 GMT, toad <[EMAIL PROTECTED]> said: > > > Hi. Reiserfs packs small files well, and has problems with large tails. > > So would it make sense to try to gzip tails before packing? > > AIX already supports this sort of compression. Interestingly enough, > they use LZ compression instead, probably for the following reasons: > > 1) Consider any patent issues - I think LZW has a problem here. gzip > I believe is free, but... gzip is free. It's LZ77 that has patent issues, and was used in GIF, AFAIR. > > 2) Remember that you're only compressing a *tail* - as such, you want > something that does a reasonable job for "very short" runs. > 'gzip < /dev/null | wc' says 20 characters of overhead - do LZ or other > schemes do better? Also, think about the startup CPU cost for each > tail on compression/decompression - is gzip optimal or are other things > better? Right, which is why you need to only write compressed data if compression actually gains you space. gzip can actually give quite good compression on small files, judging by the output of this script: (find / -size -4k -and -type f -and -size +1k | (while read x; do echo -n `ls -l "$x" | tr --squeeze " " | cut -d " " -f 5`; echo -n " "; cp -f "$x" 1.blah; rm -f 1.blah.gz; gzip -9 1.blah; echo -n `ls -l 1.blah.gz | tr --squeeze " " | cut -d " " -f 5` " "; echo "$x"; done) | less) (results are more varied for <1kb, and it doesn't matter as much).
> > Remember - this has different trade-offs than the usual usage of gzip. > Usually, you don't bother gzipping unless the file is large, and you don't > do it often, so you can afford to be slower to get better compression. > For tails, you probably want an algorithm that's 15% faster, even if the > result is 15% longer... If you're cpu-bound, you probably don't want to compress anything, unless a smaller tail is a lot easier to pack, which the reiserfs docs seem to suggest... possibly this won't gain you enough to offset the cost of compression though. Benchmarking could be interesting. For example, if you have lots of small, compressible files and good read locality, read performance could benefit from compression. OTOH, it could function as a building block for 'real compression' - a directory with an inode flag appears as a file, contains a series of small files with numbered names (or ideally without names, write the hash directly); each file is a compressed 4kb block (ordinarily this would not be stored as a tail, but here it is, after compression). This stuff probably gains greatly from a repacker and allocate on flush, and would also be cleaner with v4's plugins. As regards algorithms, e2compr has various - LZO, LZVW, gzip, and a specialised version of bzip2 (definitely not a good idea for tails :)). Also, gzip -8 is not the same as gzip -1. Although there isn't much code likely to be common with e2compr, because it would do different things. > > /Valdis > -- The road to Tycho is paved with good intentions
msg04053/pgp00000.pgp
Description: PGP signature
