Re: When substitute download + decompression is CPU-bound

Hi, Guillaume Le Vaillant skribis: > Pierre Neidhardt skribis: [...] >>> It’s not as nice as the ability to choose a download strategy, as we >>> discussed earlier, but implementing that download strategy sounds >>> tricky. >> >> If the user can choose their favourite substitute compression,

Re: When substitute download + decompression is CPU-bound

Pierre Neidhardt writes: > Nicolò Balzarotti writes: > > What wouldn't be the case? If you mean that "gzip is never the best > choice", wouldn't Zstd outperform gzip on the Raspberry Pi 1 too? My bad, you are right. Also, memory usage shoudn't be a problem. gzip uses way less (testd on ungoo

Re: When substitute download + decompression is CPU-bound

Hi, On Thu, 28 Jan 2021 at 19:07, Ludovic Courtès wrote: > We could do that. I suppose a possible agenda would be: > > 1. Start providing zstd susbstitutes anytime. However, most clients > will keep choosing lzip because it usually compresses better. > > 2. After the next release, sto

Re: When substitute download + decompression is CPU-bound

Nicolò Balzarotti skribis: > Which hardware did you use? Since you are fixing the download speed, > those results really depend on cpu speed. I ran these tests on a laptop from 2012 with an Intel i7-3630QM CPU. When the CPU speed increases, the download speed limit below which Lzip is the best

Re: When substitute download + decompression is CPU-bound

Guillaume Le Vaillant writes: > Here are a few numbers for the installation time in seconds (download > time + decompression time) when fetching 580 MB of substitutes for > download speeds between 0.5 MB/s and 20 MB/s. Which hardware did you use? Since you are fixing the download speed, those r

Re: When substitute download + decompression is CPU-bound

Pierre Neidhardt skribis: > Hi Ludo! > > Ludovic Courtès writes: > >> I suppose a possible agenda would be: >> >> 1. Start providing zstd susbstitutes anytime. However, most clients >> will keep choosing lzip because it usually compresses better. >> >> 2. After the next release, stop p

Re: When substitute download + decompression is CPU-bound

Hi Ludo! Ludovic Courtès writes: >> On Guillaume's graph, the compression speed at the default level 3 is >> about 110 MB/s, while at level 10 it's about 40 MB/s, which is >> approximately the gzip speed. >> >> If server compression time does not matter, then I agree, level >= 10 >> would be a g

Re: When substitute download + decompression is CPU-bound

Pierre Neidhardt skribis: > On Guillaume's graph, the compression speed at the default level 3 is > about 110 MB/s, while at level 10 it's about 40 MB/s, which is > approximately the gzip speed. > > If server compression time does not matter, then I agree, level >= 10 > would be a good option. >

Re: When substitute download + decompression is CPU-bound

Hi Ludo, Ludovic Courtès writes: > We could also drop gzip, but there are probably pre-1.1 daemons out > there that understand nothing but gzip¹, so perhaps that’ll have to > wait. Now, compressing substitutes three times may be somewhat > unreasonable. > > Thoughts? > Is there a request log wh

Re: When substitute download + decompression is CPU-bound

Hi Guillaume, Guillaume Le Vaillant skribis: > I compared gzip, lzip and zstd when compressing a 580 MB pack (therefore > containing "subsitutes" for several packages) with different compression > levels. Maybe the results can be of some use to someone. It’s insightful, thanks a lot! One takea

Re: When substitute download + decompression is CPU-bound

Pierre Neidhardt skribis: > Wow, impressive! :) > > Guillaume Le Vaillant writes: > >> Note that the plots only show the results using only 1 thread and > > Doesn't 1 thread defeat the purpose of parallel compression / decompression? > It was just to get a better idea of the relative compressi

Re: When substitute download + decompression is CPU-bound

I compared gzip, lzip and zstd when compressing a 580 MB pack (therefore containing "subsitutes" for several packages) with different compression levels. Maybe the results can be of some use to someone. Note that the plots only show the results using only 1 thread and standard compression levels,

Re: When substitute download + decompression is CPU-bound

Ludovic Courtès writes: > My laptop has 4 cores, so at best I’d get a 4x speedup, compared to the > 10x speedup with zstd that also comes with much lower resource usage, > etc. Of course, it's a trade off between high compression and high speed :) Since there is no universal best option, I thin

Re: When substitute download + decompression is CPU-bound

Hi Pierre, Pierre Neidhardt skribis: > Ludovic Courtès writes: > >> Well, ‘guix publish’ would first need to create multi-member archives, >> right? > > Correct, but it's trivial once the bindings have been implemented. OK. >> Also, lzlib (which is what we use) does not implement parallel >>

Re: When substitute download + decompression is CPU-bound

Looking on the Zstandard website (https://facebook.github.io/zstd/), it mentions google's snappy compression library (https://github.com/google/snappy). Snappy has some fairly good benchmarks too: Compressor Ratio Compression Decompress. zstd2.884 500 MB/s1660 M

Re: When substitute download + decompression is CPU-bound

Ludovic Courtès writes: > A lot in what sense? In terms of bandwidth usage, right? Yep, I think most of mobile data plans are still limited. Even if here in Italy is easy to get 50Gb+/monthly, I think it's not the same worldwide.

Re: When substitute download + decompression is CPU-bound

Hi, Pierre Neidhardt skribis: > Another option is plzip (parallel Lzip, an official part of Lzip). > >> decompression of ungoogled-chromium from the LAN completes in 2.4s for >> gzip vs. 7.1s for lzip. On a low-end ARMv7 device, also on the LAN, I >> get 32s (gzip) vs. 53s (lzip). > > With four

Re: When substitute download + decompression is CPU-bound

Hi, Nicolò Balzarotti skribis: > I guess this benchmark follows the distri talk, doesn't it? :) Yes, that and my own quest for optimization opportunities. :-) > File size with zstd vs zstd -9 vs current lzip: > - 71M uc.nar.lz > - 87M uc.nar.zst-9 > - 97M uc.nar.zst-default > >> Where to go f

Re: When substitute download + decompression is CPU-bound

Super interesting findings! On 14.12.20 23:20, Ludovic Courtès wrote: 2. Use Zstd like all the cool kids since it seems to have a much higher decompression speed: . 630 MB/s on ungoogled-chromium on my laptop. Woow. Not only decompression speed

Re: When substitute download + decompression is CPU-bound

Here the "pipeline" could be a CSP channel. Not sure what the term is in Guile. -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature

Re: When substitute download + decompression is CPU-bound

Nicolò Balzarotti writes: > If you download multiple files at a time, you might end up decompressing > them simultaneously. Plzip won't help then on a dual core machine, > where you might end up being cpu bound again then. Is this right? > > If it is, reducing the overall cpu usage seems to be a

Re: When substitute download + decompression is CPU-bound

Pierre Neidhardt writes: > > What do you mean? > If you download multiple files at a time, you might end up decompressing them simultaneously. Plzip won't help then on a dual core machine, where you might end up being cpu bound again then. Is this right? If it is, reducing the overall cpu usage

Re: When substitute download + decompression is CPU-bound

Pierre Neidhardt writes: > Another option is plzip (parallel Lzip, an official part of Lzip). Wouldn't that mean that this will become a problem when we'll have parallel downloads (and sometimes parallel decompression will happen)?

Re: When substitute download + decompression is CPU-bound

Another option is plzip (parallel Lzip, an official part of Lzip). > decompression of ungoogled-chromium from the LAN completes in 2.4s for > gzip vs. 7.1s for lzip. On a low-end ARMv7 device, also on the LAN, I > get 32s (gzip) vs. 53s (lzip). With four cores, plzip would beat gzip in the first

Re: When substitute download + decompression is CPU-bound

Ludovic Courtès writes: > Hi Guix! > Hi Ludo > Quick decompression bench: I guess this benchmark follows the distri talk, doesn't it? :) File size with zstd vs zstd -9 vs current lzip: - 71M uc.nar.lz - 87M uc.nar.zst-9 - 97M uc.nar.zst-default > Where to go from here? Several options: >

Re: When substitute download + decompression is CPU-bound

My proposed changes to allow for parallel download assume downloads are network-bound, so they can be separate from other jobs. If downloads are actually CPU-bound, then it has indeed no merit at all :) Le 14 décembre 2020 17:20:17 GMT-05:00, "Ludovic Courtès" a écrit : >Hi Guix! > >Consider t

When substitute download + decompression is CPU-bound

Hi Guix! Consider these two files: https://ci.guix.gnu.org/nar/gzip/kfcrrl6p6f6v51jg5rirmq3q067zxih6-ungoogled-chromium-87.0.4280.88-0.b78cb92 https://ci.guix.gnu.org/nar/lzip/kfcrrl6p6f6v51jg5rirmq3q067zxih6-ungoogled-chromium-87.0.4280.88-0.b78cb92 Quick decompression bench: --8<---