Hi, On Oct 8, 2016, at 4:09, scsijon <[email protected]> wrote:
> On 10/05/2016 08:39 PM, René Rebe wrote: >> Hey, >> >> On Oct 5, 2016, at 7:56, René Rebe <[email protected]> wrote: >> >>> Hi, >>> >>> On Oct 5, 2016, at 0:58, scsijon <[email protected]> wrote: >>> >>>> On 10/05/2016 08:51 AM, René Rebe wrote: >>>>> Hey there, >>>>> >>>>> I recently was made aware of Zstandard compression, and like some of it’s >>>>> properties. >>>>> For example blazing fast decompression speed. This always annoyed me the >>>>> most with the bzip2 that we like for two decades or so. >>>>> But this should also save storage and download traffic. >>>>> >>>>> We prepared trunk so far, and we are re-compressing the trunk mirror >>>>> archive. >>>>> >>>>> I estimate some fallout, such as the host build system need to have zstd >>>>> and it’s file need to know the zstd magic (patch in T2). >>>>> >>>>> I will also test switching the “iso” binaries packages to zstd by default >>>>> for smaller iso image and blazingly fast unpacking (e.g. to solid state >>>>> storage, …). >>>>> >>>>> Cheers, >>>>> René >>>>> >>>> >>>> Hopefully if your going to do that, you will still allow settings to be >>>> made for the current tar.bz2 input source files and output compiled >>>> packages to stay as is as well as your new z compression. I really don't >>>> want to have to download all of the mirror source packages I use again to >>>> start with just because of a compression change. >>> >>> Well one archive files in 18 years or so, … - scripts are complex enough - >>> rather do not over engineer them more and keep it small and simple. >>> You can once run something like: >>> >>> find download -name "*bz2" | while read f; do echo "$f" ${f%.bz2}.zst ; [ >>> -f "${f%.bz2}.zst" ] || bunzip2 < $f | zstd -19 > ${f%.bz2}.zst ; done >>> >>>> However, I do like the idea of having the compiled packages in a different >>>> format to the source tree packages as well as a smaller iso to deal with, >>>> i'd rather do that so I instantaneously know what has been built. >>> >>> Actually we already supported other binary output compressors, like .gz, >>> .lzo, .xz, and .lzma or so. >>> >>>> Can you please give us a link to which of the z compressions you intend to >>>> use (I think there are 3 at present being touted to become the >>>> 'standard'). Also are you also going to change the kernel setting to match >>>> so compiled commands can be left compressed until used? That would shrink >>>> things a bit although the processir will have to do a little bit more work. >>> >>> Not sure what you mean - I will do nothing to the kernel though. >>> >>> Actually zstandard is much faster than all most other decompression - >>> especially bzip2 (it is like as fast as just computing the fiel md5!) ;-) >>> >>> https://blog.fefe.de/?ts=a90c6783 >>> http://zstd.net >> > > well this is a new one to me and from a quick read doesn't seem to be kernel > linked though so that makes four z standards. I wonder which will win. Why does someone need to win? We choose whatever suits us best today, 150 other compression standards are fine too. Maybe one day another becomes better than that, ... > My understanding of the kernel setting, is that it will allow the commands, > etc. over a certain size (anything non text format and the like) to remain in > a compressed format until needed and only uncompressed when in ram. i'm not > sure if this one is still alpha test mode or up to beta testing in the kernel > end yet as I haven't both checked what's happening for a while on the kernel > boards for a few months or attempted to compile a kernel since early 4.x. This has nothing todo with kernel settings - this is about our binar package tar.bz2 on the iso. If I wanted a compressed files system I’d use btrfs with compress=whateever. >> How much better actually is it? t2/trunk x86-i486: >> >> -rw-r--r-- 1 root root 48064172 Oct 5 09:37 gcc-5.3.0.tar.zst.ultra >> -rw-r--r-- 1 root root 65978259 Oct 5 09:08 gcc-5.3.0.tar.zst.19 >> -rw-r--r-- 1 root root 82942235 Oct 5 09:33 gcc-5.3.0.tar.bz2 >> >> Actually I thought to use level -19 for T2 packagers, but seeing --ultra -22 >> maybe we should use that (though it takes a while to compress, … ;-) >> > > Also, if you use the ultra compression I do wonder if people will be turned > off with a new download set time to repackage before download command > compleats. Maybe you need to do a comparison test for a fresh download and > with the two sets available ask if the second longer time is ok considering > the space available. However on the other side of it with 'drive' space > becoming cheaper and 'drives' becoming seriously larger, i'm not sure that is > something to be considered in the equation any more. I can not really follow this text - what do you mean to say / ask? Normal users are only positively affected by this, the .zstd files are smaller than the .bz2 we used, and de-compress way faster. It is a win-win all over - the only thing that takes longer is compression - which usually only developers do. Using our mirrors the downloads already come down as .zstd, ... And even than you can use the parallel pzstd flavor on a multi-core machine to dramatically decrease the wall-clock-time. René -- ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
----------------------------------------------------------- If you wish to unsubscribe from this mailing, send mail to [email protected] with a subject of: unsubscribe t2
