Re: Increase in CPU usage in 8.7.1 vs 8.6.0 for rate-limited downloads

[Stefan Eissing via curl-library]

> Am 15.05.2024 um 08:56 schrieb David Pfitzner via curl-library 
> <curl-library@lists.haxx.se>:
> 
> Hi,
> 
> I've noticed that from curl 8.6.0 to 8.7.1, the CPU usage when doing 
> rate-limited downloads with libcurl has increased significantly. This is most 
> noticeable when making multiple (rate-limited) downloads in parallel, for 
> large files. (I've seen this in a custom application using libcurl, and then 
> reproduced using the 'curl' tool.)
> 
> For example, running 'curl' with the following (where the file a32 contains 
> 32 copies of " -o /dev/null http://127.0.0.1:9999/speed10";, a test resource 
> with size=46450490):
> 
> #curl-8.6.0:
> for x in 1 2 3 4 5 ; do /usr/bin/time ../curl-8_6_0 --parallel 
> --parallel-immediate --no-progress-meter --limit-rate 10485760 `cat ./a32` ; 
> done
>         5.02 real         0.20 user         0.67 sys
>         4.95 real         0.15 user         0.68 sys
>         5.02 real         0.25 user         0.63 sys
>         5.00 real         0.21 user         0.63 sys
>         4.49 real         0.23 user         0.62 sys
> 
> # curl-8.7.1:
> for x in 1 2 3 4 5 ; do /usr/bin/time ../curl-8_7_1 --parallel 
> --parallel-immediate --no-progress-meter --limit-rate 10485760 `cat ./a32` ; 
> done
>         4.51 real         0.52 user         0.65 sys
>         5.01 real         0.55 user         0.63 sys
>         5.02 real         0.56 user         0.59 sys
>         4.96 real         0.42 user         0.59 sys
>         5.01 real         0.37 user         0.71 sys
> 
> Notice the increase in user-space CPU usage (almost doubled). (Of course the 
> actual effect can vary depending on many factors - the number of parallel 
> downloads, the specified rate limit, file sizes, network speed, etc.)
> 
> From testing, this change appears to be due to this commit:
>   https://github.com/curl/curl/commit/db5c9f4f9e0779b49624752b135281a0717b277b
> 
> That change was to make rate limiting more accurate, and perhaps the increase 
> in CPU usage is a necessary (and expected?) cost of doing so. But, in my use 
> case the rate limiting accuracy in 8.6.0 was "good enough", and I would 
> prefer not to incur the increase in CPU usage seen in 8.7.1. But, I'm not 
> sure how to do that. Perhaps it would be useful for a user of libcurl to be 
> able to (somehow) control this tradeoff between rate-limiting accuracy and 
> CPU usage?

Thanks for the data. I believe there are certainly improvements possible here. 
We are much in favour of making transfers not only work correctly but also 
efficiently. It is more a matter of someone knowledgable finding the time to 
work on this.

Cheers,
Stefan

> 
> Regards,
> David Pfitzner
> -- 
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette:   https://curl.se/mail/etiquette.html

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html