Hey there, double checked decompression performance on aging outdated silicon for you:
root@oqo:/usr/src/t2-trunk/download/mirror/l# time zstd -d < linux-4.7.tar.zst > /dev/null real 0m33.215s user 0m32.264s sys 0m0.836s root@oqo:/usr/src/t2-trunk/download/mirror/l# time bzip2 -d < linux-4.7.tar.bz2 > /dev/null real 2m25.235s user 2m23.204s sys 0m1.692s this is on a really slooooow 999MHz Transmeta Crusoe: http://t2-project.org/hardware/portable/OQO/01+/ vendor_id : GenuineTMx86 cpu family : 5 model : 4 model name : Transmeta(tm) Crusoe(tm) Processor TM5800 stepping : 3 cpu MHz : 999.015 cache size : 512 KB fdiv_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr cx8 sep cmov mmx longrun lrti constant_tsc eagerfpu bugs : bogomips : 1998.03 clflush size : 32 cache_alignment : 32 On Oct 8, 2016, at 11:06, René Rebe <[email protected]> wrote: > Hi, > > On Oct 8, 2016, at 4:09, scsijon <[email protected]> wrote: > >> On 10/05/2016 08:39 PM, René Rebe wrote: >>> Hey, >>> >>> On Oct 5, 2016, at 7:56, René Rebe <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> On Oct 5, 2016, at 0:58, scsijon <[email protected]> wrote: >>>> >>>>> On 10/05/2016 08:51 AM, René Rebe wrote: >>>>>> Hey there, >>>>>> >>>>>> I recently was made aware of Zstandard compression, and like some of >>>>>> it’s properties. >>>>>> For example blazing fast decompression speed. This always annoyed me the >>>>>> most with the bzip2 that we like for two decades or so. >>>>>> But this should also save storage and download traffic. >>>>>> >>>>>> We prepared trunk so far, and we are re-compressing the trunk mirror >>>>>> archive. >>>>>> >>>>>> I estimate some fallout, such as the host build system need to have zstd >>>>>> and it’s file need to know the zstd magic (patch in T2). >>>>>> >>>>>> I will also test switching the “iso” binaries packages to zstd by >>>>>> default for smaller iso image and blazingly fast unpacking (e.g. to >>>>>> solid state storage, …). >>>>>> >>>>>> Cheers, >>>>>> René >>>>>> >>>>> >>>>> Hopefully if your going to do that, you will still allow settings to be >>>>> made for the current tar.bz2 input source files and output compiled >>>>> packages to stay as is as well as your new z compression. I really don't >>>>> want to have to download all of the mirror source packages I use again to >>>>> start with just because of a compression change. >>>> >>>> Well one archive files in 18 years or so, … - scripts are complex enough - >>>> rather do not over engineer them more and keep it small and simple. >>>> You can once run something like: >>>> >>>> find download -name "*bz2" | while read f; do echo "$f" ${f%.bz2}.zst ; [ >>>> -f "${f%.bz2}.zst" ] || bunzip2 < $f | zstd -19 > ${f%.bz2}.zst ; done >>>> >>>>> However, I do like the idea of having the compiled packages in a >>>>> different format to the source tree packages as well as a smaller iso to >>>>> deal with, i'd rather do that so I instantaneously know what has been >>>>> built. >>>> >>>> Actually we already supported other binary output compressors, like .gz, >>>> .lzo, .xz, and .lzma or so. >>>> >>>>> Can you please give us a link to which of the z compressions you intend >>>>> to use (I think there are 3 at present being touted to become the >>>>> 'standard'). Also are you also going to change the kernel setting to >>>>> match so compiled commands can be left compressed until used? That would >>>>> shrink things a bit although the processir will have to do a little bit >>>>> more work. >>>> >>>> Not sure what you mean - I will do nothing to the kernel though. >>>> >>>> Actually zstandard is much faster than all most other decompression - >>>> especially bzip2 (it is like as fast as just computing the fiel md5!) ;-) >>>> >>>> https://blog.fefe.de/?ts=a90c6783 >>>> http://zstd.net >>> >> >> well this is a new one to me and from a quick read doesn't seem to be kernel >> linked though so that makes four z standards. I wonder which will win. > > Why does someone need to win? We choose whatever suits us best today, 150 > other compression standards are fine too. Maybe one day another becomes > better than that, ... > >> My understanding of the kernel setting, is that it will allow the commands, >> etc. over a certain size (anything non text format and the like) to remain >> in a compressed format until needed and only uncompressed when in ram. i'm >> not sure if this one is still alpha test mode or up to beta testing in the >> kernel end yet as I haven't both checked what's happening for a while on the >> kernel boards for a few months or attempted to compile a kernel since early >> 4.x. > > This has nothing todo with kernel settings - this is about our binar package > tar.bz2 on the iso. > > If I wanted a compressed files system I’d use btrfs with compress=whateever. > >>> How much better actually is it? t2/trunk x86-i486: >>> >>> -rw-r--r-- 1 root root 48064172 Oct 5 09:37 gcc-5.3.0.tar.zst.ultra >>> -rw-r--r-- 1 root root 65978259 Oct 5 09:08 gcc-5.3.0.tar.zst.19 >>> -rw-r--r-- 1 root root 82942235 Oct 5 09:33 gcc-5.3.0.tar.bz2 >>> >>> Actually I thought to use level -19 for T2 packagers, but seeing --ultra >>> -22 maybe we should use that (though it takes a while to compress, … ;-) >>> >> >> Also, if you use the ultra compression I do wonder if people will be turned >> off with a new download set time to repackage before download command >> compleats. Maybe you need to do a comparison test for a fresh download and >> with the two sets available ask if the second longer time is ok considering >> the space available. However on the other side of it with 'drive' space >> becoming cheaper and 'drives' becoming seriously larger, i'm not sure that >> is something to be considered in the equation any more. > > I can not really follow this text - what do you mean to say / ask? > > Normal users are only positively affected by this, the .zstd files are > smaller than the .bz2 we used, and de-compress way faster. > It is a win-win all over - the only thing that takes longer is compression - > which usually only developers do. Using our mirrors the downloads already > come down as .zstd, ... > And even than you can use the parallel pzstd flavor on a multi-core machine > to dramatically decrease the wall-clock-time. > > René > > -- > ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin > http://exactcode.com | http://exactscan.com | http://ocrkit.com | > http://t2-project.org | http://rene.rebe.de > > ----------------------------------------------------------- > If you wish to unsubscribe from this mailing, send mail to > [email protected] with a subject of: unsubscribe t2 -- ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
----------------------------------------------------------- If you wish to unsubscribe from this mailing, send mail to [email protected] with a subject of: unsubscribe t2
