> My point is that just the same statement: "there are cases that look like as > the rate limit hasn’t applied at all" - is also true for 8.6.0. The only > difference is that with 8.6.0 you have to go to smaller transfer sizes to see > the effect.
No, this is not true. As I mentioned in my message, in 8.6.0, the smaller transfer sizes decreased accuracy, but it was always applied unless the transfers were extremely small. But in 8.18.0 we have cases when no rate limit is applied to the transfers depending on run-time conditions. > So yes, at a given transfer size the behavior may be different, because the > rate-limiting "resolution" has changed, but claiming that this is a radical > change in behavior seems wrong. It is a radical change, because we now have cases in 8.18.0, when the rate limit is either not applied at all or there is a huge gap between the rate limit and the measured transfer speed. And it may increase the operational costs significantly for libcurl users in such cases. Let’s consider an example from my test where the rate limit wasn’t applied: Url=… file_1M.bin, max_speed=20000000 bps time=18 ms, dnld=1048576 B, speed=457518843 bps, spd_diff=437518843 bps, pct=2187.6 % If I do multiple downloads like this back-to-back (i.e. 100+ times) then CPU utilization for these transfers will be as high as without the rate limit both for client and the server. But in 8.6.0 – where the rate limit for these conditions was applied correctly, the total CPU utilization (both for client and server) was a few times lower. So, while 8.18.0 rate limiting mechanism decreased CPU utilization compared to 8.6.0 for the cases where it works, it increased CPU utilization on order of magnitude both for client and server for run-time conditions where it doesn’t work. And this may have a large cost effect for applications with large population using cloud services. And what makes things worse is that it is now very difficult to predict where the new mechanism will “not work”. In the same test example, the rate limit of 16 Mbps was applied correctly: Url=… file_1M.bin, max_speed=16000000 bps time=510 ms, dnld=1048576 B, speed=16423165 bps, spd_diff=423165 bps, pct=2.6 % This means that in 8.18.0 some client app doing multiple back-to-back transfers, may suddenly step on cases where the rate limiting doesn’t work as before and thus incur much higher operational costs. And this kind of defeats the purpose of optimizing the rate limiting mechanism to use less CPU cycles. I am trying to avoid calling it “regression”, but it is a radical behavior change, which potentially can cost much more to libcurl users than previous implementation. Thanks, Dmitry From: David Pfitzner <[email protected]> Sent: Tuesday, January 6, 2026 7:36 PM To: Dmitry Karpov <[email protected]> Cc: libcurl development <[email protected]> Subject: Re: [EXTERNAL] Re: Rate limit regressions in libcurl 8.18.0 vs 8.17.0 On Wed, Jan 7, 2026 at 1:02 PM Dmitry Karpov <[email protected]<mailto:[email protected]>> wrote: In the 8.6.0, the rate limiter mechanism was kind of embedded into the run loop, so it was always engaged and showed very good precision (< 5% deviation of the measured speed from the rate limit) even for relatively small transfer sizes (~300 KB) and super-fast network speeds. It worked properly even when client and server were on the same host machine where there were no network delays. Here is the test example for 390 KB download throttling test on 8.6.0 when server and client are on the same machine: (8.6.0) Speed limit test [server on the same host as client], iterations=5 Url=http://192.168.1.3/Data/DSC_391kB.jpg, max_speed=8000000 bps time=399 ms, dnld=400096 B, speed=8020969 bps, spd_diff=20969 bps, pct=0.3 % time=405 ms, dnld=400096 B, speed=7884597 bps, spd_diff=-115403 bps, pct=-1.4 % time=406 ms, dnld=400096 B, speed=7882480 bps, spd_diff=-117520 bps, pct=-1.5 % time=404 ms, dnld=400096 B, speed=7910005 bps, spd_diff=-89995 bps, pct=-1.1 % time=397 ms, dnld=400096 B, speed=8056036 bps, spd_diff=56036 bps, pct=0.7 % -------------------------------------------------------------------- avg_deviation=-0.6 %, max_deviation=-1.5 % Url=http://192.168.1.3/Data/DSC_391kB.jpg, max_speed=16000000 bps time=209 ms, dnld=400096 B, speed=15286666 bps, spd_diff=-713334 bps, pct=-4.5 % time=197 ms, dnld=400096 B, speed=16231980 bps, spd_diff=231980 bps, pct=1.4 % time=197 ms, dnld=400096 B, speed=16202236 bps, spd_diff=202236 bps, pct=1.3 % time=198 ms, dnld=400096 B, speed=16110410 bps, spd_diff=110410 bps, pct=0.7 % time=195 ms, dnld=400096 B, speed=16394185 bps, spd_diff=394185 bps, pct=2.5 % -------------------------------------------------------------------- avg_deviation=0.3 %, max_deviation=-4.5 % Url=http://192.168.1.3/Data/DSC_391kB.jpg, max_speed=20000000 bps time=162 ms, dnld=400096 B, speed=19697882 bps, spd_diff=-302118 bps, pct=-1.5 % time=161 ms, dnld=400096 B, speed=19856004 bps, spd_diff=-143996 bps, pct=-0.7 % time=166 ms, dnld=400096 B, speed=19249034 bps, spd_diff=-750966 bps, pct=-3.8 % time=162 ms, dnld=400096 B, speed=19721549 bps, spd_diff=-278451 bps, pct=-1.4 % time=159 ms, dnld=400096 B, speed=20035479 bps, spd_diff=35479 bps, pct=0.2 % -------------------------------------------------------------------- avg_deviation=-1.4 %, max_deviation=-3.8 % Url=http://192.168.1.3/Data/DSC_391kB.jpg, max_speed=32000000 bps time=97 ms, dnld=400096 B, speed=32785684 bps, spd_diff=785684 bps, pct=2.5 % time=99 ms, dnld=400096 B, speed=32143324 bps, spd_diff=143324 bps, pct=0.4 % time=99 ms, dnld=400096 B, speed=32290545 bps, spd_diff=290545 bps, pct=0.9 % time=98 ms, dnld=400096 B, speed=32509298 bps, spd_diff=509298 bps, pct=1.6 % time=98 ms, dnld=400096 B, speed=32478290 bps, spd_diff=478290 bps, pct=1.5 % -------------------------------------------------------------------- avg_deviation=1.4 %, max_deviation=2.5 % The precision started deteriorating (> 15%) only for the downloads < 250 KB, but the measured transfer speeds were always less than the rate limit, so, the back-to-back transfers never created high load issues on the server side. So, to be on the safe side, my statement about high precision of rate limiting in 8.6.0 should be corrected as "for any transfer sizes > 1MB and any network conditions". Unfortunately, the price for such a precision in 8.6.0 was higher CPU utilization as more cycles were needed to calculate and apply delays. So, the intent to minimize CPU usage while losing some precision in 8.18 is totally understandable, but the loss of precision due to lesser points of delay calculations should be somehow compensated at the end of the transfer to avoid regressions for applications that expect more or less reliable correlation between applied rate limit and measured transfer speeds. Unfortunately, in 8.18.0 there are cases that look like as the rate limit hasn’t applied at all, and it creates issues for logic making decisions based on transfer speed measurements, and unexpected higher load on the server side. My point is that just the same statement: "there are cases that look like as the rate limit hasn’t applied at all" - is also true for 8.6.0. The only difference is that with 8.6.0 you have to go to smaller transfer sizes to see the effect. So it depends on what transfer sizes you are interested in (and other parameters). So yes, at a given transfer size the behavior may be different, because the rate-limiting "resolution" has changed, but claiming that this is a radical change in behavior seems wrong. -- David From: curl-library <[email protected]<mailto:[email protected]>> On Behalf Of David Pfitzner via curl-library Sent: Tuesday, January 6, 2026 5:17 PM To: libcurl development <[email protected]<mailto:[email protected]>> Cc: David Pfitzner <[email protected]<mailto:[email protected]>> Subject: [EXTERNAL] Re: Rate limit regressions in libcurl 8.18.0 vs 8.17.0 On Wed, Jan 7, 2026 at 10:05 AM Dmitry Karpov via curl-library <[email protected]<mailto:[email protected]>> wrote: The 8.6.0 worked very well for rate limiting as it provided stable and predictable measured transfer download speeds for any transfer sizes and network conditions. That "for any transfer sizes and network conditions" is not true. The rate limiting may have done what you wanted for the parameters you care about, or the parameters that you happened to test, but even for 8.6.0 there are possible cases where rate limiting is specified but has no effect. (That is, because the transfer finishes before any rate limiting delay is imposed.). -- David
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
