I hear that you have a very specific application scenario where the curl 8.6.0 "almost" (your words) helped you, but the versions after that are no longer that "almost". Especially for cases where the overall network transfer is less than a second (or just a few).
The curl rate limit works in "bytes per second". It does not say when *during* the second the bytes are received. The bytes may be received in the first few nanoseconds. If the transfer has not ended by that, libcurl will wait the remainder of the second before doing the next receive, obeying the limit. This is in accordance to what you want, except at the end of a transfer. Here, our definitions of what "rate limit" means differ. (And I would really prefer you not calling our view on things as "not working" or a "regression". It is working fine as we define rate limits. It is just not what you want. Respect each other's views.) An example: a transfer of 100 bytes is configured with a rate limit of 1000 bytes/s. - In your definition, the transfer should complete exactly after 100ms. - In our definition, the transfer completes when the 100 bytes have been received, never delivering more than 1000 bytes in a second. And there was no second that it delivered more. For longer transfers, this all does not matter much as the last second has decreasing impact. Now, besides your libcurl application, there are other applications that use rate limits but want to have transfers reported as complete when all data has arrived. I would hate my steam downloads to pause at the end, for example. You have a specific application case and your proposed solution is incompatible with other cases. I do not know your application and cannot judge how to best solve that. It seems to be in need to some "holding queue" where finished libcurl transfers are held to idle the last second remainder before the application acts on the completion. This is what you are asking libcurl to implement, so your application does not have to. tl;dr The current rate limit implementation is what should be natural for many libcurl applications. For longer transfers, it works for you was well. For short ones, unfortunately, it does not give you the precision that you want in your application. Acknowledged. Adding a separate rate limit implementation in libcurl, so your application does not have to, is not a convincing argument. - Stefan > Am 05.01.2026 um 21:29 schrieb Dmitry Karpov via curl-library > <[email protected]>: > >> I don't think libcurl should do that. Once the transfer is complete, I think >> it should return/say so. > > But that creates a big problem for back-to-back transfers like it was > observed in my examples, for which it looked like there was no rate limit > applied at all. > And it creates a load problem on the servers which don't see the rate > limiting and must serve data with higher load. > > As far as I know, CPU throttling mechanisms add delays after some operation > is completed to prevent scheduling of next operations > even though it may not be needed for the already completed operation. > > And here we have a kind of the same case - we need to add some delay after > the transfer to complete to prevent performing the next transfer too soon > because we don't have an "ideal" rate limiting mechanism and need to > compensate for that after the transfer is done. > > In 8.6.0, we had almost "ideal" rate limiting mechanism where the speed > measurements were done more frequently for the price of higher CPU > utilization. > But this allowed to perform transfers with rate limiting applied more > smoothly and with very high precision. > > So, if we decreased the number of speed measurements to reduce CPU usage, > then we would need to compensate the loss of precision by adding some delays > at the end. > > Otherwise, we can have run-time conditions (like in my test cases) where in > some multi-transfer use cases the rate limiting is not actually working, > and it will be a regression for clients expecting it to work (and it used to > work in 8.17, although not with the same precision as in earlier releases > like in 8.6.0). > >> If your app thinks it needs to add that extra wait, it is really easy for >> you to add a sleep there though. > > Unfortunately, it is not easy. > My app is a very large multi-layered framework, where libcurl transfers are > used in too many components, including modules closed for modifications, > to make it feasible to add additional code which adds delays after each > transfer if the rate limit is not observed by the libcurl rate limiting > option. > And I think I am not alone who has applications like mine. > > If client code needs to do that kind of actions, then it kind of defeats the > purpose of the libcurl rate limit option making it unreliable and > unpredictable. > And because it will be a regression from the earlier releases, not sure that > many folks will be happy about it and the perspective > to add additional code to work around the new problem. > > Thanks! > Dmitry > > > -----Original Message----- > From: Daniel Stenberg <[email protected]> > Sent: Saturday, January 3, 2026 2:36 PM > To: Dmitry Karpov via curl-library <[email protected]> > Cc: Dmitry Karpov <[email protected]> > Subject: [EXTERNAL] Re: Rate limit regressions in libcurl 8.18.0 vs 8.17.0 > > On Wed, 31 Dec 2025, Dmitry Karpov via curl-library wrote: > >> As I discussed it with Stefan before, the rate limit mechanism should >> apply some small delay at the end of a throttled transfer to maintain >> the specified rate limit for the transfer. > > I don't think libcurl should do that. Once the transfer is complete, I think > it should return/say so. > >> And even though it will not save the bandwidth as all the data has >> been already transferred, it will provide proper network speed >> measurements for the decision making logic (i.e. bitrate selection >> mechanism in video streaming apps) and will help to decrease server >> load in back-to-back multi-transfer scenarios. > > If your app thinks it needs to add that extra wait, it is really easy for you > to add a sleep there though. > > -- > > / daniel.haxx.se || https://rock-solid.curl.dev > -- > Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library > Etiquette: https://curl.se/mail/etiquette.html -- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
