On 10/05/2016 08:39 PM, René Rebe wrote:
Hey,
On Oct 5, 2016, at 7:56, René Rebe <[email protected]> wrote:
Hi,
On Oct 5, 2016, at 0:58, scsijon <[email protected]> wrote:
On 10/05/2016 08:51 AM, René Rebe wrote:
Hey there,
I recently was made aware of Zstandard compression, and like some of it’s
properties.
For example blazing fast decompression speed. This always annoyed me the most
with the bzip2 that we like for two decades or so.
But this should also save storage and download traffic.
We prepared trunk so far, and we are re-compressing the trunk mirror archive.
I estimate some fallout, such as the host build system need to have zstd and
it’s file need to know the zstd magic (patch in T2).
I will also test switching the “iso” binaries packages to zstd by default for
smaller iso image and blazingly fast unpacking (e.g. to solid state storage, …).
Cheers,
René
Hopefully if your going to do that, you will still allow settings to be made
for the current tar.bz2 input source files and output compiled packages to stay
as is as well as your new z compression. I really don't want to have to
download all of the mirror source packages I use again to start with just
because of a compression change.
Well one archive files in 18 years or so, … - scripts are complex enough -
rather do not over engineer them more and keep it small and simple.
You can once run something like:
find download -name "*bz2" | while read f; do echo "$f" ${f%.bz2}.zst ; [ -f
"${f%.bz2}.zst" ] || bunzip2 < $f | zstd -19 > ${f%.bz2}.zst ; done
However, I do like the idea of having the compiled packages in a different
format to the source tree packages as well as a smaller iso to deal with, i'd
rather do that so I instantaneously know what has been built.
Actually we already supported other binary output compressors, like .gz, .lzo,
.xz, and .lzma or so.
Can you please give us a link to which of the z compressions you intend to use
(I think there are 3 at present being touted to become the 'standard'). Also
are you also going to change the kernel setting to match so compiled commands
can be left compressed until used? That would shrink things a bit although the
processir will have to do a little bit more work.
Not sure what you mean - I will do nothing to the kernel though.
Actually zstandard is much faster than all most other decompression -
especially bzip2 (it is like as fast as just computing the fiel md5!) ;-)
https://blog.fefe.de/?ts=a90c6783
http://zstd.net
well this is a new one to me and from a quick read doesn't seem to be
kernel linked though so that makes four z standards. I wonder which will
win.
My understanding of the kernel setting, is that it will allow the
commands, etc. over a certain size (anything non text format and the
like) to remain in a compressed format until needed and only
uncompressed when in ram. i'm not sure if this one is still alpha test
mode or up to beta testing in the kernel end yet as I haven't both
checked what's happening for a while on the kernel boards for a few
months or attempted to compile a kernel since early 4.x.
How much better actually is it? t2/trunk x86-i486:
-rw-r--r-- 1 root root 48064172 Oct 5 09:37 gcc-5.3.0.tar.zst.ultra
-rw-r--r-- 1 root root 65978259 Oct 5 09:08 gcc-5.3.0.tar.zst.19
-rw-r--r-- 1 root root 82942235 Oct 5 09:33 gcc-5.3.0.tar.bz2
Actually I thought to use level -19 for T2 packagers, but seeing --ultra -22
maybe we should use that (though it takes a while to compress, … ;-)
Also, if you use the ultra compression I do wonder if people will be
turned off with a new download set time to repackage before download
command compleats. Maybe you need to do a comparison test for a fresh
download and with the two sets available ask if the second longer time
is ok considering the space available. However on the other side of it
with 'drive' space becoming cheaper and 'drives' becoming seriously
larger, i'm not sure that is something to be considered in the equation
any more.
René
scsijon
-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to
[email protected] with a subject of: unsubscribe t2
