Re: Slow Upload Performance on High-Bandwidth connections on windows
On Thu, 7 Feb 2013, Christian Hägele wrote: Optimizing for very high bandwidth (over 100MBit/s) might need some additional work on the internals of curl which are out of scope of the issue you described. I agree. But we could start with documenting the results and suggestions somwhere, of what you can do to maximize transfer performance. Like for example setting SO_SNDBUF to several megabytes on windows - if that truly helps. I'm very interested if the speed-problem you described initially when doing cross country transfers over the internet is solved by the patch you wrote. I am too! -- / daniel.haxx.se--- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
Hello I finally received access to slow upload machine config. First of all: the ping value is 57ms (it's even worse than i thought), netsh interface tcp global values are the same as described in previous messages. I made several curl builds: with modified CURL_MAX_WRITE_SIZE (MAX) and modified set SO_SNDBUF routine, and run quick test by upload 70mb. MAX=524288 SNDBUF=ORIGINAL ROUTINE sftp:1129k ftp:5700k MAX=524288 SNDBUF=DETECTOS (disable adj for post Vista) sftp:1013k ftp:5675k MAX=16384 SNDBUF=8388608 sftp:272k ftp:4950k MAX=16384 SNDBUF=4194304 sftp:270k ftp:6130k MAX=16384 SNDBUF=2097152 sftp:245k ftp:6247k MAX=16384 SNDBUF=524288 sftp:269k ftp:6681k MAX=16384 SNDBUF=262144 sftp:242k ftp:4815k MAX=16384 SNDBUF=131072 sftp:270k ftp:2410k MAX=16384 SNDBUF=65536 sftp:245k ftp:987k MAX=16384 SNDBUF=32768 sftp:246k ftp:604k MAX=16384 SNDBUF=ORIGINAL ROUTINE sftp:249k ftp:549k MAX=16384 SNDBUF=DETECTOS (disable adj for post Vista) sftp:250k ftp:560k so disabled SO_SNDBUF adjustment did not fix this problem. may be curl-lib should allow to specify buffer size(with CURLOPT_BUFFERSIZE or similar opt) larger than CURL_MAX_WRITE_SIZE? default value will remain equal to CURL_MAX_WRITE_SIZE(16kb) but command line tool or host app will receive ability to increase that buffer without recompilation. ps ignore sftp values 08.02.2013, в 1:10, Daniel Stenberg dan...@haxx.se написал(а): On Wed, 6 Feb 2013, Kurushin Andrey wrote: 1. the main bug report (filezilla vs far manager vs curl) contains test of cross-countries test results (ping 8ms). recent tests (with custom builds of 7.28.1) were made inside single subnet (ping 1ms) so they have better speed values. Well, 8 vs 1 ms latency shouldn't make much of a difference to a fairly large plain FTP transfer. 2. CURL_MAX_WRITE_SIZE = 16kb and SO_SNDBUF=8mb gives very good ftp upload speed (equal to version with big global buffer) Interesting! So have you tried with for example 4mb, 2mb or 512kb etc to see if there's a particular cut-off limit somewhere? It is a bit weird that the TCP stack can't sort this out by itself... but sftp upload speed still have 6Mb/s limit. Yes, but I've explained several times on this list and on the libssh2 list, the reason for SFTP's problems is _entirely_ different and in fact just growing the buffer size is only one piece of the fix to the SFTP performance bottleneck we can see. If you really want to address the SFTP problem then I won't mind helping out, but we cannot mix the two problems into one and pretending that they are the same. So the best way to increase both ftp and sftp upload speed is to increase CURL_MAX_WRITE_SIZE; Possibly. But we can't unconditionally just bump the 16K up to something crazy like 512K. We have applications who do hundreds of even thousands of parallel transfers, they will not appreciate such a massive memory bloat. Possibly the time has come to start working on allowing a custom size for the buffer. -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
On Wed, 6 Feb 2013, Kurushin Andrey wrote: 1. the main bug report (filezilla vs far manager vs curl) contains test of cross-countries test results (ping 8ms). recent tests (with custom builds of 7.28.1) were made inside single subnet (ping 1ms) so they have better speed values. Well, 8 vs 1 ms latency shouldn't make much of a difference to a fairly large plain FTP transfer. 2. CURL_MAX_WRITE_SIZE = 16kb and SO_SNDBUF=8mb gives very good ftp upload speed (equal to version with big global buffer) Interesting! So have you tried with for example 4mb, 2mb or 512kb etc to see if there's a particular cut-off limit somewhere? It is a bit weird that the TCP stack can't sort this out by itself... but sftp upload speed still have 6Mb/s limit. Yes, but I've explained several times on this list and on the libssh2 list, the reason for SFTP's problems is _entirely_ different and in fact just growing the buffer size is only one piece of the fix to the SFTP performance bottleneck we can see. If you really want to address the SFTP problem then I won't mind helping out, but we cannot mix the two problems into one and pretending that they are the same. So the best way to increase both ftp and sftp upload speed is to increase CURL_MAX_WRITE_SIZE; Possibly. But we can't unconditionally just bump the 16K up to something crazy like 512K. We have applications who do hundreds of even thousands of parallel transfers, they will not appreciate such a massive memory bloat. Possibly the time has come to start working on allowing a custom size for the buffer. -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
Kurushin Andrey ajax16384 at gmail.com writes: 1. the main bug report (filezilla vs far manager vs curl) contains test of cross-countries test results (ping 8ms). recent tests (with custom builds of 7.28.1) were made inside single subnet (ping 1ms) so they have better speed values. Doing transfers over the internet is a whole other story than doing it in a LAN. I think you are addressing 2 different issues here. Optimizing for very high bandwidth (over 100MBit/s) might need some additional work on the internals of curl which are out of scope of the issue you described. I'm very interested if the speed-problem you described initially when doing cross country transfers over the internet is solved by the patch you wrote. By solved I mean getting the same or better performance than FileZilla or similar tools. If the problem is not solved I can't really imagine what the problem is, because the Windows TCP-Stack should increase the SO_SNBUF itself as necessary. That's why the manual adjustment of SO_SNDBUF should only have a very minor effect. Regards, Christian --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
1. the main bug report (filezilla vs far manager vs curl) contains test of cross-countries test results (ping 8ms). recent tests (with custom builds of 7.28.1) were made inside single subnet (ping 1ms) so they have better speed values. 2. CURL_MAX_WRITE_SIZE = 16kb and SO_SNDBUF=8mb gives very good ftp upload speed (equal to version with big global buffer), but sftp upload speed still have 6Mb/s limit. So the best way to increase both ftp and sftp upload speed is to increase CURL_MAX_WRITE_SIZE; I understand that sftp limit may have different nature - but now it's also depends on CURL_MAX_WRITE_SIZE value. 06.02.2013, в 0:17, Christian Hägele haeg...@teamviewer.com написал(а): Thank you for your input. That rules out some misconfiguration in windows. In my opinion CURL_MAX_WRITE_SIZE should not have a big effect on the speed, because the TCP-stack shouldn't care in what chunks the application wants to send data over the TCP-connection. It can have an effect if the application doesn't fill up the TCP-buffer fast enough, though. So i have 2 questions as follow up: 1. In the bug-report you said you are seeing upload-speeds of ~500KiB/s with curl and up to 6MiB/s with FileZilla. The other numbers you provided recently where somewhere around 18MiB/s. So I suppose you did not test the linux ftp server in another country? What where your test-setup for your recent numbers? Especially the latency (round-trip-time) between the two peers is very important here. 2. Did you try keep the CURL_MAX_WRITE_SIZE at default and try to increase the SO_SNDBUF option manually to a very large value (e.g. 8MiB)? I have the assumption that in your latest tests the limiting factor was not the TCP-connection but the CPU in pumping the data fast enough to the TCP-stack. If that is the case then that is another issue. Regards, Christian --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
On Tue, 5 Feb 2013, Pierre Joye wrote: But yes, it needs improvements here. I can't think of any better way to solve this than to detect OS level at run-time and only SNDBUF if = windows 2008, so that Vista and later won't get it changed. It is possible to detect at runtime the OS version and set the right set of options. I do that in PHP as well for some features supporte only in recent windows version. Would it be something acceptable in curl? Only the 1st call would be slower (fetching the OS info). Yes I think we need to do something like that! -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
hi! On Tue, Feb 5, 2013 at 11:05 AM, Daniel Stenberg dan...@haxx.se wrote: On Tue, 5 Feb 2013, Pierre Joye wrote: But yes, it needs improvements here. I can't think of any better way to solve this than to detect OS level at run-time and only SNDBUF if = windows 2008, so that Vista and later won't get it changed. It is possible to detect at runtime the OS version and set the right set of options. I do that in PHP as well for some features supporte only in recent windows version. Would it be something acceptable in curl? Only the 1st call would be slower (fetching the OS info). Yes I think we need to do something like that! ok, will write a patch tonight, but won't have time to test the socket part, sadly. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
Hello According to my investigation at http://sourceforge.net/p/curl/bugs/1188/#ca3a skipping of SO_SNDBUF adjustment does not solve low performance. Thanks Andrei 05.02.2013, в 19:50, Pierre Joye pierre@gmail.com написал(а): hi! On Tue, Feb 5, 2013 at 11:05 AM, Daniel Stenberg dan...@haxx.se wrote: On Tue, 5 Feb 2013, Pierre Joye wrote: But yes, it needs improvements here. I can't think of any better way to solve this than to detect OS level at run-time and only SNDBUF if = windows 2008, so that Vista and later won't get it changed. It is possible to detect at runtime the OS version and set the right set of options. I do that in PHP as well for some features supporte only in recent windows version. Would it be something acceptable in curl? Only the 1st call would be slower (fetching the OS info). Yes I think we need to do something like that! ok, will write a patch tonight, but won't have time to test the socket part, sadly. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
Курушин Андрей ajax16384 at gmail.com writes: According to my investigation at http://sourceforge.net/p/curl/bugs/1188/#ca3a skipping of SO_SNDBUF adjustment does not solve low performance. I looked at your patch and that solved my issue with slow performance. I did not try to change the CURL_MAX_WRITE_SIZE, yet. It would be interesting to have the output of the following command on an elevated command prompt an the sending side (Windows2008R2): netsh interface tcp show global The values for receive side window scaling should be normal and and the value for congestion control provider should be ctcp. If the values are different please to try set them an redo your tests. Regards, Christian --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
win2k8r2 settings during previous tests: TCP Global Parameters -- Receive-Side Scaling State : enabled Chimney Offload State : automatic NetDMA State: enabled Direct Cache Acess (DCA): disabled Receive Window Auto-Tuning Level: normal Add-On Congestion Control Provider : ctcp ECN Capability : disabled RFC 1323 Timestamps : disabled 05.02.2013, в 22:46, Christian Hägele haeg...@teamviewer.com написал(а): netsh interface tcp show global --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
Thank you for your input. That rules out some misconfiguration in windows. In my opinion CURL_MAX_WRITE_SIZE should not have a big effect on the speed, because the TCP-stack shouldn't care in what chunks the application wants to send data over the TCP-connection. It can have an effect if the application doesn't fill up the TCP-buffer fast enough, though. So i have 2 questions as follow up: 1. In the bug-report you said you are seeing upload-speeds of ~500KiB/s with curl and up to 6MiB/s with FileZilla. The other numbers you provided recently where somewhere around 18MiB/s. So I suppose you did not test the linux ftp server in another country? What where your test-setup for your recent numbers? Especially the latency (round-trip-time) between the two peers is very important here. 2. Did you try keep the CURL_MAX_WRITE_SIZE at default and try to increase the SO_SNDBUF option manually to a very large value (e.g. 8MiB)? I have the assumption that in your latest tests the limiting factor was not the TCP-connection but the CPU in pumping the data fast enough to the TCP-stack. If that is the case then that is another issue. Regards, Christian --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
On Tue, 5 Feb 2013, Christian Hägele wrote: So i have 2 questions as follow up: Just a quick addition from me as well: stick with comparing plain (unencrypted FTP) to begin with so that we know it is plain and simple socket usage we play with. SFTP is an *entirely* different beast and for that there's a set of known tricks we can use (but currently aren't) to improve speed. -- / daniel.haxx.se--- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
On Fri, 1 Feb 2013, Christian Hägele wrote: The problem described in http://support.microsoft.com/kb/823764 doesn't seems to apply to Win-Vista. So do you mean Curl_sendbufset() should not be set for Vista/2008 R2? If so, there should be an adaption to the running Windows version. Try it; ref. curl_sspi.c. I encountered the problem described in the KB-articem yself once. I did some research on the issue, but unfortunately I don't remember everything. Well, we got reports in the past showing problems unless we followed the advice. Clearly we're now facing an issue that needs to be applied in some cases but is downright wrong in other situations. How lovely. As nobody knows the reason for that workaround exactly I would suggest to make an option to set SO_SNDBUF. The problem with that is that nobody would ever know when to set that option or not. You basically would just have to experience the problem and then at run-time fairly arbitrarily try the next transfer with the option set. It seems like a pretty awful way really... That could also work on all OSes which support the socket option SO_SNDBUF. We already have a callback option that allows the application to set socket options so this can already be done. The point with the SNDBUF modification libcurl does is that Microsoft clearly says there's a risk for problems otherwise and it is libcurl's job to try to act as good as possible to work everywhere in an optimized way. But yes, it needs improvements here. I can't think of any better way to solve this than to detect OS level at run-time and only SNDBUF if = windows 2008, so that Vista and later won't get it changed. On all modern OSes this should be the best choice. Right. Unless the vendor says it isn't. =) -- / daniel.haxx.se--- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
On Mon, 4 Feb 2013, Christian Hägele wrote: You are right that this is ugly if you were dependent on that workaround. However, I believe that not many people are experience these problems. I don't know when the CURL_MAX_WRITE_SIZE was lowered from 20Kib to 16KiB, but there is a chance that the problem only occurred with a send size of 20KiB. No. We decided to set the size to 16K way back in the age of dawn, long before we had any SNDBUF work-arounds introduced. Alas, we didn't get the SNDBUF problems with 20K, we got (and worked around) them with 16K buffers. 1. Get rid of setting the SO_SNDBUF-option to 16416 via Curl_sndbufset-function unless you are running Windows older than Vista. Personally I would like to get rid of this completely, but I understand you don't want to break anything for existing users. Ack. I would like to see this done. 2. Add command-line options for the curl-command-line-tool for setting SO_SNDBUF and SO_RCVBUF. That would be good because if you are running a OS that doesn't do a good job of determine the right buffer-size that options could improve transfer- speed a lot. (e.g. Linux pre 2.6.16) I don't think this is strictly needed. We never had any reports of problems on these and we did uploads long before 2.6.16 was born and we've had many users use that or older linux kernels as well. I think we can leave that until someone actually wants or needs it. I already started to write the changes for no 2). I have a question for that: Right now I implemented this via the CURLOPT_SOCKOPTFUNCTION function. However, some other options (e.g. TCP_NODELAY) have their own option in libCurl. Has this any special reason? A simple reason: stuff that we can assume to be commonly requested or used by applications get to have their own option so that not every single user of libcurl would have to implement the same thing. -- / daniel.haxx.se--- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
Christian Hägele haeg...@teamviewer.com wrote: The reason for that is the Curl_sndbufset-function in connect.c. On windows the socket-sendbuffer is set to a value of CURL_MAX_WRITE_SIZE + 32 (16416 bytes). There is a also a strange comment why this is done. When the SO_SNDBUF is not explicitly set the socket-sendbuffer is increased automatically by the OS (starting with windows Vista/Server 2008). Did you also see the comment in include/curl/curl.h? 'CURL_MAX_WRITE_SIZE' can be set a compile-time. I suggest you do that to not make transfer slower for people using an older Windows (Win-XP). The problem described in http://support.microsoft.com/kb/823764 doesn't seems to apply to Win-Vista. So do you mean Curl_sendbufset() should not be set for Vista/2008 R2? If so, there should be an adaption to the running Windows version. Try it; ref. curl_sspi.c. --gv --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Slow Upload Performance on High-Bandwidth connections on windows
Am 01.02.2013, 13:47 Uhr, schrieb Gisle Vanem gva...@broadpark.no: Did you also see the comment in include/curl/curl.h? 'CURL_MAX_WRITE_SIZE' can be set a compile-time. I suggest you do that to not make transfer slower for people using an older Windows (Win-XP). The size of 16KiB seems to be a good choice for most situations. As this value has to be set at compile time I wouldn't touch it. The problem described in http://support.microsoft.com/kb/823764 doesn't seems to apply to Win-Vista. So do you mean Curl_sendbufset() should not be set for Vista/2008 R2? If so, there should be an adaption to the running Windows version. Try it; ref. curl_sspi.c. I encountered the problem described in the KB-articem yself once. I did some research on the issue, but unfortunately I don't remember everything. But what I remember was that the problem only occurs in very special circumstances. In that case it was the chunk size of 80KiB which reproduced the issue. Maybe the issue only occurred when the CURL_MAX_WRITE_SIZE was set to 20 KiB? Does someone know? I wouldn't do a special workaround depending on the OS during runtime. Such workarounds often lead to more problems in the future and it makes the code worse. As nobody knows the reason for that workaround exactly I would suggest to make an option to set SO_SNDBUF. That could also work on all OSes which support the socket option SO_SNDBUF. Many other programs have similar command line options. But for the standard-case I wouldn't touch the SO_SNDBUF-size at all and let the OS do its magic (that's what currently done for SO_RCVBUF). On all modern OSes this should be the best choice. Any opinions an that? Christian --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
