On 4/20/12, Daniel Stenberg <dan...@haxx.se> wrote: > On Thu, 19 Apr 2012, Adam Craig wrote: > >> I found that, with the remote server, while larger packet sizes yielded >> some >> improvement in speed, it was only barely noticeable compared to the >> fluctuations in speed due to outside factors. On the local network, the >> improvement in speed is pronounced from 2K to 20K but levels off at higher >> >> sizes. > > Thanks for doing this! > > Unfortunately I still believe there are too many unknowns in this equation > and > you've done your tests on a too limited test set to make us draw very many > conclusions for libssh2 genericly as opposed for libssh2 in your particular > case. >
My apologies. I do not have the time or resources right now to do a more comprehensive study. Hopefully I will when I take care of some of my other standing commitments. In the mean time, I will study the RFC and libssh2 source code more carefully. > For example: > > A) you only used OpenSSH servers on Linux, possibly not even using very > different versions. we must expect different server ends to react > differently > > B) what were the RTT times against these servers? SFTP will perform > significantly different for varying RTT. One of the major challanges > with > SFTP is to make it run fast for the whole range of possible RTT > Again, I must apologize. The remote Red Hat server has become permanently unavailable, so I cannot get an RTT measurement for it. Ubuntu Server on local network: PING 10.0.0.16 (10.0.0.16): 56 data bytes 64 bytes from 10.0.0.16: icmp_seq=0 ttl=64 time=3.821 ms 64 bytes from 10.0.0.16: icmp_seq=1 ttl=64 time=2.786 ms 64 bytes from 10.0.0.16: icmp_seq=2 ttl=64 time=3.268 ms 64 bytes from 10.0.0.16: icmp_seq=3 ttl=64 time=2.501 ms 64 bytes from 10.0.0.16: icmp_seq=4 ttl=64 time=2.616 ms 64 bytes from 10.0.0.16: icmp_seq=5 ttl=64 time=4.580 ms 64 bytes from 10.0.0.16: icmp_seq=6 ttl=64 time=3.031 ms 64 bytes from 10.0.0.16: icmp_seq=7 ttl=64 time=5.368 ms 64 bytes from 10.0.0.16: icmp_seq=8 ttl=64 time=3.966 ms 64 bytes from 10.0.0.16: icmp_seq=9 ttl=64 time=3.210 ms 64 bytes from 10.0.0.16: icmp_seq=10 ttl=64 time=4.215 ms 64 bytes from 10.0.0.16: icmp_seq=11 ttl=64 time=6.010 ms 64 bytes from 10.0.0.16: icmp_seq=12 ttl=64 time=2.678 ms 64 bytes from 10.0.0.16: icmp_seq=13 ttl=64 time=2.765 ms 64 bytes from 10.0.0.16: icmp_seq=14 ttl=64 time=2.799 ms 64 bytes from 10.0.0.16: icmp_seq=15 ttl=64 time=2.788 ms 64 bytes from 10.0.0.16: icmp_seq=16 ttl=64 time=2.998 ms 64 bytes from 10.0.0.16: icmp_seq=17 ttl=64 time=3.258 ms 64 bytes from 10.0.0.16: icmp_seq=18 ttl=64 time=3.044 ms 64 bytes from 10.0.0.16: icmp_seq=19 ttl=64 time=2.806 ms 64 bytes from 10.0.0.16: icmp_seq=20 ttl=64 time=2.652 ms 64 bytes from 10.0.0.16: icmp_seq=21 ttl=64 time=2.678 ms 64 bytes from 10.0.0.16: icmp_seq=22 ttl=64 time=3.037 ms > C) which crypto backend and version was used in libssh2? I've seen people > report very different results when using gcrypt vs openssl libssh2 version: 1.4.0-20120229 Since I did not specify gcrypt, it should be using openssl. OpenSSL version: 0.9.8r 8 Feb 2011, which came with the Macbook. It seems to have some customizations unique to OS X, like automatically using the OS X keychain in stead of requiring a trust store from a file or directory. I have not been able to find a complete list of differences between the openssl that ships with OS X and the standard version. > > D) how large buffers did you pass in to libssh2_sftp_read() when you ran > these > tests and how did did different such sizes affect the test results? > I allocated a buffer of size 524288, about twice the size of the largest file I needed to upload. The stated sizes I passed in to the libssh2_sftp_read function were the same as the max chunk sizes listed in the spreadsheet. >> I noticed the following comment on the line defining the limit on >> upload chunk size: >> /* >> * MAX_SFTP_OUTGOING_SIZE MUST not be larger than 32500 or so. This is the >> * amount of data sent in each FXP_WRITE packet >> */ >> #define MAX_SFTP_OUTGOING_SIZE 32500 >> >> My guess is the "MUST" is due to the following, found in libssh2_priv.h: > > It is due to phrasing in the SSH and SFTP protocol specs, yes. > >> I also noticed the following in sftp.c: >> /* This is the maximum packet length to accept, as larger than this >> indicate >> some kind of server problem. */ >> #define LIBSSH2_SFTP_PACKET_MAXLEN 80000 >> >> I do not see why 80K is the cutoff here. > > It is no magic "cutoff" really, it is just a way for libssh2 to detect > problems as if it gets a very large packet size it is a sign that something > is > wrong and it should escape rather than trying to allocate memory for it. > >> If the spec says the max packet size is 35K, then it would make more sense >> >> to make all the hard-coded limits on packet size 35K > > Then read the spec. Does it say 35K? And further, does 35K work against all > the relevant server implementations? We're being slightly conservative in > the > name of interoperability. > This was a mistake on my part. According to http://tools.ietf.org/html/rfc4253#section-6.1, 35K is the maximum total packet size. The uncompressed data payload should only be up to 32768 bytes. Thus, 32768 should work against any spec-compliant server implementations. Have you encountered many cases where it has needed to be smaller? >> and let users limit packet size further through the buffer sizes they pass >> >> to the read and write statements. > > Then you have grossly missed the finer implementation details of the SFTP > read > and write functions libssh2 provides. It does a lot of work to allow > applications to get high throughput by splitting up the buffer in many > slices. > (which is what I alluded to above in my point "D"). Leaving that to the > application would certainly be possible, but will require a new API to still > offer high speed transfers. > Apparently I have misunderstood something. I will look into it, but it may be a while, since I have other aspects of my current project that still need a lot of work. Thank you again, Adam Craig > -- > > / daniel.haxx.se > _______________________________________________ > libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel > _______________________________________________ libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel
