Hey, On Oct 5, 2016, at 7:56, René Rebe <[email protected]> wrote:
> Hi, > > On Oct 5, 2016, at 0:58, scsijon <[email protected]> wrote: > >> On 10/05/2016 08:51 AM, René Rebe wrote: >>> Hey there, >>> >>> I recently was made aware of Zstandard compression, and like some of it’s >>> properties. >>> For example blazing fast decompression speed. This always annoyed me the >>> most with the bzip2 that we like for two decades or so. >>> But this should also save storage and download traffic. >>> >>> We prepared trunk so far, and we are re-compressing the trunk mirror >>> archive. >>> >>> I estimate some fallout, such as the host build system need to have zstd >>> and it’s file need to know the zstd magic (patch in T2). >>> >>> I will also test switching the “iso” binaries packages to zstd by default >>> for smaller iso image and blazingly fast unpacking (e.g. to solid state >>> storage, …). >>> >>> Cheers, >>> René >>> >> >> Hopefully if your going to do that, you will still allow settings to be made >> for the current tar.bz2 input source files and output compiled packages to >> stay as is as well as your new z compression. I really don't want to have to >> download all of the mirror source packages I use again to start with just >> because of a compression change. > > Well one archive files in 18 years or so, … - scripts are complex enough - > rather do not over engineer them more and keep it small and simple. > You can once run something like: > > find download -name "*bz2" | while read f; do echo "$f" ${f%.bz2}.zst ; [ -f > "${f%.bz2}.zst" ] || bunzip2 < $f | zstd -19 > ${f%.bz2}.zst ; done > >> However, I do like the idea of having the compiled packages in a different >> format to the source tree packages as well as a smaller iso to deal with, >> i'd rather do that so I instantaneously know what has been built. > > Actually we already supported other binary output compressors, like .gz, > .lzo, .xz, and .lzma or so. > >> Can you please give us a link to which of the z compressions you intend to >> use (I think there are 3 at present being touted to become the 'standard'). >> Also are you also going to change the kernel setting to match so compiled >> commands can be left compressed until used? That would shrink things a bit >> although the processir will have to do a little bit more work. > > Not sure what you mean - I will do nothing to the kernel though. > > Actually zstandard is much faster than all most other decompression - > especially bzip2 (it is like as fast as just computing the fiel md5!) ;-) > > https://blog.fefe.de/?ts=a90c6783 > http://zstd.net How much better actually is it? t2/trunk x86-i486: -rw-r--r-- 1 root root 48064172 Oct 5 09:37 gcc-5.3.0.tar.zst.ultra -rw-r--r-- 1 root root 65978259 Oct 5 09:08 gcc-5.3.0.tar.zst.19 -rw-r--r-- 1 root root 82942235 Oct 5 09:33 gcc-5.3.0.tar.bz2 Actually I thought to use level -19 for T2 packagers, but seeing --ultra -22 maybe we should use that (though it takes a while to compress, … ;-) René -- ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
----------------------------------------------------------- If you wish to unsubscribe from this mailing, send mail to [email protected] with a subject of: unsubscribe t2
