Correction about buffering. I think I took for “buffering” the socket read buffer size, which is different from the internal read buffer controlled by the CURLOPT_BUFFERSIZE option.
I played with the CURLOPT_BUFFERSIZE option, and I can tell that it affects the speed deviations. The bigger the CURLOPT_BUFFERSIZE value, the bigger the differences. (Probably, because the throttling is applied after this internal buffer is fully processed) Even 7.84 with 2MB buffer size (I increased the max from 512KB to 2MB) has significant speed deviations. But what is strange is that the default value should be the same for both 8.4 and 7.84 = 16KB, and with the default CURLOPT_BUFFERSIZE value they should behave the same. But when I explicitly set CURLOPT_BUFFERSIZE to 16KB in master/8.4 the deviations don’t go down to the 7.84 values, and still are quite big. And larger CURLOPT_BUFFERSIZE values in master/8.4 have much bigger effect on speed deviations than in 7.84. From: curl-library <curl-library-boun...@lists.haxx.se> On Behalf Of Dmitry Karpov via curl-library Sent: Friday, December 1, 2023 4:03 PM To: libcurl development <curl-library@lists.haxx.se> Cc: Dmitry Karpov <dkar...@roku.com> Subject: [EXTERNAL] Re: Large download speed deviations for CURLOPT_MAX_RECV_SPEED_LARGE in 8.4.0 No, buffering is the same. I explicitly set 1MB socket receive buffer to avoid any default buffering differences between 8.4. and 7.84. The deviations are not sensitive to file sizes. They slightly vary between file sizes, but still are quite large for all file sizes. For example: Speed limit test [LAN (~700 mbps)] **** 50 MB **** Url=http://192.168.0.6/Data/file_50M.bin, max_speed=8000000 bps time=63451 ms, dnld=52428800 B, speed=6610292 bps, spd_diff=-1389708 bps, pct=-17.4 % time=63601 ms, dnld=52428800 B, speed=6594698 bps, spd_diff=-1405302 bps, pct=-17.6 % time=64145 ms, dnld=52428800 B, speed=6538777 bps, spd_diff=-1461223 bps, pct=-18.3 % Url=http://192.168.0.6/Data/file_50M.bin, max_speed=16000000 bps time=34411 ms, dnld=52428800 B, speed=12188838 bps, spd_diff=-3811162 bps, pct=-23.8 % time=34331 ms, dnld=52428800 B, speed=12217023 bps, spd_diff=-3782977 bps, pct=-23.6 % **** 10 MB **** Url=http://192.168.0.6/Data/file_10M.bin, max_speed=8000000 bps time=12958 ms, dnld=10485760 B, speed=6473605 bps, spd_diff=-1526395 bps, pct=-19.1 % time=12865 ms, dnld=10485760 B, speed=6520179 bps, spd_diff=-1479821 bps, pct=-18.5 % Url=http://192.168.0.6/Data/file_10M.bin, max_speed=16000000 bps time=6759 ms, dnld=10485760 B, speed=12409340 bps, spd_diff=-3590660 bps, pct=-22.4 % time=6670 ms, dnld=10485760 B, speed=12576294 bps, spd_diff=-3423706 bps, pct=-21.4 % … Also, the size values in the “write function” are the same for both 8.4. and 7.84. And total transfer times and speeds were measured by libcurl correctly and match what I measured myself using timers, so it doesn’t look like a performance issue. The factor which affects the deviations most is a difference between the network speed and the speed limit. The less the difference the less the deviations. This makes me think that probably current transfer speed evaluation (needed to calculate a transfer pause time to meet the speed limit) somehow regressed in 8.4 (or earlier) and lost the precision it had in 7.84. I can’t run tests on Linux at this moment or do a more advanced debugging as you suggested, but I don’t think that the timing calculations really differ between Windows or Linux. If I don’t use a speed limit, then I don’t see any performance differences between 8.4. and 7.84 for local network transfers, so, I guess it is just timing calculations that need to be debugged. I will try to spend more time debugging it next week, but at least on Windows this issue is quite visible with my test settings. (local network (700 Mbps, 100 Mbps, 20 Mbps speeds), and speed limits (8 Mbps and 16 Mbps) ). I tried multiple local network setups and different Windows clients and saw the same issue. Thanks, Dmitry Karpov From: curl-library <curl-library-boun...@lists.haxx.se<mailto:curl-library-boun...@lists.haxx.se>> On Behalf Of Timothe Litt via curl-library Sent: Friday, December 1, 2023 1:56 AM To: curl-library@lists.haxx.se<mailto:curl-library@lists.haxx.se> Cc: Timothe Litt <l...@acm.org<mailto:l...@acm.org>> Subject: [EXTERNAL] Re: Large download speed deviations for CURLOPT_MAX_RECV_SPEED_LARGE in 8.4.0 On 01-Dec-23 02:32, Dmitry Karpov via curl-library wrote: In my tests, I downloaded a 5 MB file from some local network server using different network speeds. The test client was running on Windows (mentioning it, just in case). Random thought. Symptoms sound like different buffering. 5MB isn't all that big, what with all the buffering and today's network speeds. To get some clues: (a) is the deviation sensitive to file size? e.g., same deviation with 50MB? (b) do you see the same deviations from a Linux client? - if so, does strace show different setsockopt calls, perhaps setting buffer sizes? (c) Are you using the CURLOPT_WRITEFUNCTION? If so, is it getting the same 'nmemb' values? (if not, you can add one that does an fwrite() & log the nmemb values...) (d) If you enable VERBOSE and DEBUGFUNCTION and log to a (preallocated to minimize timing) memory buffer and print at exit, and the deviations don't go away: what's different in the log? (e) Does the actual elapsed time (measured by the OS/application) differ significantly? If so, it's not a timing measurement issue; rather it's a performance issue. In all the above "different" means between libcurl versions. Timothe Litt ACM Distinguished Engineer -------------------------- This communication may not represent the ACM or my employer's views, if any, on the matters discussed.
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
