Hi,
On Oct 5, 2016, at 0:58, scsijon <[email protected]> wrote:
> On 10/05/2016 08:51 AM, René Rebe wrote:
>> Hey there,
>>
>> I recently was made aware of Zstandard compression, and like some of it’s
>> properties.
>> For example blazing fast decompression speed. This always annoyed me the
>> most with the bzip2 that we like for two decades or so.
>> But this should also save storage and download traffic.
>>
>> We prepared trunk so far, and we are re-compressing the trunk mirror archive.
>>
>> I estimate some fallout, such as the host build system need to have zstd and
>> it’s file need to know the zstd magic (patch in T2).
>>
>> I will also test switching the “iso” binaries packages to zstd by default
>> for smaller iso image and blazingly fast unpacking (e.g. to solid state
>> storage, …).
>>
>> Cheers,
>> René
>>
>
> Hopefully if your going to do that, you will still allow settings to be made
> for the current tar.bz2 input source files and output compiled packages to
> stay as is as well as your new z compression. I really don't want to have to
> download all of the mirror source packages I use again to start with just
> because of a compression change.
Well one archive files in 18 years or so, … - scripts are complex enough -
rather do not over engineer them more and keep it small and simple.
You can once run something like:
find download -name "*bz2" | while read f; do echo "$f" ${f%.bz2}.zst ; [ -f
"${f%.bz2}.zst" ] || bunzip2 < $f | zstd -19 > ${f%.bz2}.zst ; done
> However, I do like the idea of having the compiled packages in a different
> format to the source tree packages as well as a smaller iso to deal with, i'd
> rather do that so I instantaneously know what has been built.
Actually we already supported other binary output compressors, like .gz, .lzo,
.xz, and .lzma or so.
> Can you please give us a link to which of the z compressions you intend to
> use (I think there are 3 at present being touted to become the 'standard').
> Also are you also going to change the kernel setting to match so compiled
> commands can be left compressed until used? That would shrink things a bit
> although the processir will have to do a little bit more work.
Not sure what you mean - I will do nothing to the kernel though.
Actually zstandard is much faster than all most other decompression -
especially bzip2 (it is like as fast as just computing the fiel md5!) ;-)
https://blog.fefe.de/?ts=a90c6783
http://zstd.net
René
--
ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
DE Legal: Amtsgericht Berlin (Charlottenburg) HRB 105123B, Tax-ID#: DE251602478
Managing Director: René Rebe
http://exactcode.com | http://exactscan.com | http://ocrkit.com |
http://t2-project.org | http://rene.rebe.de
-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to
[email protected] with a subject of: unsubscribe t2
