Re: cURL uploads 0 length file for sftp transfer.
Curl under cygwin - TCP keepalive I need to download a file from a remote server via a https request, and the remote server takes several minutes to prepare the file. Without any keepalives, the remote end will terminate the socket, but it doesn't close it (which I know is weird). Running curl with --keepalive 45 on a linux centos7 machine, enables the transfer to work. With tcpdump, I see the nop packet every 45 seconds. But on cygwin, the --keepalive parameter doesn't seem to have effect. Running wireshark, I don't see any keepalives. Running the request in a browser, chrome or firefox, works fine, and I can see keepalives in wireshark, so clearly they are either setting the socket options correctly, or they are sending their own nop. Running on a vm, vmware workstation / centos8 , on the same machine where I am running curl, I can see the keepalives. I have downloaded and rebuilt curl and that did not resolve the issue. Can someone shed some light on this? On Sat, Sep 7, 2019 at 9:39 PM ALbert via cygwin wrote: > I upgraded to the latest version curl 7.65.3 with libcurl 7.65.3, same > issue. > curl 7.65.3 (i686-pc-cygwin) libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.11 > brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4) > libssh/0.8.7/openssl/zlib nghttp2/1.37.0 > Thanks. > > On Friday, September 6, 2019, 3:44:13 PM EDT, ALbert < > albert...@yahoo.com> wrote: > > We recently upgraded cURL from version:curl 7.50.3 (i686-pc-cygwin) > libcurl/7.50.3 OpenSSL/1.0.2j zlib/1.2.8 libidn/1.29 libpsl/0.14.0 > (+libidn/1.29) libssh2/1.7.0 nghttp2/1.14.0tocurl 7.64.1 (i686-pc-cygwin) > libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 > libpsl/0.21.0 (+libidn2/2.0.4) libssh/0.8.7/openssl/zlib nghttp2/1.37.0 > Since the upgrade, we've seen problems uploading text files to sftp > servers. Here's the call that we make: > curl--trace-ascii trace.txt -k sftp://:@/file.txt-T > file.txt > > On initial call, a 0-length file is uploaded, with successful return code. > On immediate rerun, a full file is uploaded (if we wait few minutes, > 0-length file will get uploaded). Below are the traces from initial and > subsequent calls. We also tried using same version curl on a linux box, > without any issues. We also downloaded latest version of curl from curl for > Windows, "curl 7.65.3 (i386-pc-win32) libcurl/7.65.3 OpenSSL/1.1.1c > (Schannel) zlib/1.2.11 brotli/1.0.7 WinIDN libssh2/1.9.0 nghttp2/1.39.2", > and its also working correctly. > Unfortunately, the server we're uploading to belongs to a client, and we > can't see if this is an issue on a server side. However, even if it was, > why would linux curl and windows curl (from curl.haxx.se) would work > correctly? > Any help or pointers would be appreciated. Thank you. > > Here's the trace files: > From initial call:== Info: STATE: INIT => CONNECT handle 0x80068340; line > 1356 (connection #-5000) > == Info: Added connection 0. The cache now contains 1 members > == Info: STATE: CONNECT => WAITRESOLVE handle 0x80068340; line 1397 > (connection #0) > == Info: Trying XXX.XX.XX.XX...== Info: TCP_NODELAY set > == Info: STATE: WAITRESOLVE => WAITCONNECT handle 0x80068340; line 1476 > (connection #0) > == Info: Connected to XXX ( XXX.XX.XX.XX) port 22 (#0)== Info: STATE: > WAITCONNECT => SENDPROTOCONNECT handle 0x80068340; line 1532 (connection #0) > == Info: Marked for [keep alive]: SSH default > == Info: User: Opus_efg > == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_INIT (line 2115) > == Info: SSH 0x80048f90 state change from SSH_INIT to SSH_S_STARTUP (line > 582) > == Info: STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x80068340; line > 1547 (connection #0) > == Info: SSH 0x80048f90 state change from SSH_S_STARTUP to SSH_HOSTKEY > (line 595) > == Info: SSH 0x80048f90 state change from SSH_HOSTKEY to SSH_AUTHLIST > (line 605) > == Info: SSH 0x80048f90 state change from SSH_AUTHLIST to > SSH_AUTH_PKEY_INIT (line 628) > == Info: Authentication using SSH public key file > == Info: SSH 0x80048f90 state change from SSH_AUTH_PKEY_INIT to > SSH_AUTH_KEY_INIT (line 697) > == Info: SSH 0x80048f90 state change from SSH_AUTH_KEY_INIT to > SSH_AUTH_KEY (line 743) > == Info: completed keyboard interactive authentication > == Info: SSH 0x80048f90 state change from SSH_AUTH_KEY to SSH_AUTH_DONE > (line 761) > == Info: Authentication complete > == Info: SSH 0x80048f90 state change from SSH_AUTH_DONE to SSH_SFTP_INIT > (line 807) > == Info: SSH 0x80048f90 state change from SSH_SFTP_INIT to > SSH_SFTP_REALPATH (line 833) > == Info: SSH CONNECT phase done > == Info: SSH 0x80048f90 state change from SSH_SFTP_REALPATH to SSH_STOP > (line 850) > == Info: STATE: PROTOCONNECT => DO handle 0x80068340; line 1566 > (connection #0) > == Info: DO phase starts > == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_SFTP_QUOTE_INIT > (line 2332) > == Info: SSH 0x80048f90 state change from SSH_SFTP_QUOTE_INIT to > SSH_SFTP_GETINFO (line 868) >
Re: cURL uploads 0 length file for sftp transfer.
I upgraded to the latest version curl 7.65.3 with libcurl 7.65.3, same issue. curl 7.65.3 (i686-pc-cygwin) libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4) libssh/0.8.7/openssl/zlib nghttp2/1.37.0 Thanks. On Friday, September 6, 2019, 3:44:13 PM EDT, ALbert wrote: We recently upgraded cURL from version:curl 7.50.3 (i686-pc-cygwin) libcurl/7.50.3 OpenSSL/1.0.2j zlib/1.2.8 libidn/1.29 libpsl/0.14.0 (+libidn/1.29) libssh2/1.7.0 nghttp2/1.14.0tocurl 7.64.1 (i686-pc-cygwin) libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4) libssh/0.8.7/openssl/zlib nghttp2/1.37.0 Since the upgrade, we've seen problems uploading text files to sftp servers. Here's the call that we make: curl--trace-ascii trace.txt -k sftp://:@/file.txt-T file.txt On initial call, a 0-length file is uploaded, with successful return code. On immediate rerun, a full file is uploaded (if we wait few minutes, 0-length file will get uploaded). Below are the traces from initial and subsequent calls. We also tried using same version curl on a linux box, without any issues. We also downloaded latest version of curl from curl for Windows, "curl 7.65.3 (i386-pc-win32) libcurl/7.65.3 OpenSSL/1.1.1c (Schannel) zlib/1.2.11 brotli/1.0.7 WinIDN libssh2/1.9.0 nghttp2/1.39.2", and its also working correctly. Unfortunately, the server we're uploading to belongs to a client, and we can't see if this is an issue on a server side. However, even if it was, why would linux curl and windows curl (from curl.haxx.se) would work correctly? Any help or pointers would be appreciated. Thank you. Here's the trace files: >From initial call:== Info: STATE: INIT => CONNECT handle 0x80068340; line 1356 >(connection #-5000) == Info: Added connection 0. The cache now contains 1 members == Info: STATE: CONNECT => WAITRESOLVE handle 0x80068340; line 1397 (connection #0) == Info: Trying XXX.XX.XX.XX...== Info: TCP_NODELAY set == Info: STATE: WAITRESOLVE => WAITCONNECT handle 0x80068340; line 1476 (connection #0) == Info: Connected to XXX ( XXX.XX.XX.XX) port 22 (#0)== Info: STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x80068340; line 1532 (connection #0) == Info: Marked for [keep alive]: SSH default == Info: User: Opus_efg == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_INIT (line 2115) == Info: SSH 0x80048f90 state change from SSH_INIT to SSH_S_STARTUP (line 582) == Info: STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x80068340; line 1547 (connection #0) == Info: SSH 0x80048f90 state change from SSH_S_STARTUP to SSH_HOSTKEY (line 595) == Info: SSH 0x80048f90 state change from SSH_HOSTKEY to SSH_AUTHLIST (line 605) == Info: SSH 0x80048f90 state change from SSH_AUTHLIST to SSH_AUTH_PKEY_INIT (line 628) == Info: Authentication using SSH public key file == Info: SSH 0x80048f90 state change from SSH_AUTH_PKEY_INIT to SSH_AUTH_KEY_INIT (line 697) == Info: SSH 0x80048f90 state change from SSH_AUTH_KEY_INIT to SSH_AUTH_KEY (line 743) == Info: completed keyboard interactive authentication == Info: SSH 0x80048f90 state change from SSH_AUTH_KEY to SSH_AUTH_DONE (line 761) == Info: Authentication complete == Info: SSH 0x80048f90 state change from SSH_AUTH_DONE to SSH_SFTP_INIT (line 807) == Info: SSH 0x80048f90 state change from SSH_SFTP_INIT to SSH_SFTP_REALPATH (line 833) == Info: SSH CONNECT phase done == Info: SSH 0x80048f90 state change from SSH_SFTP_REALPATH to SSH_STOP (line 850) == Info: STATE: PROTOCONNECT => DO handle 0x80068340; line 1566 (connection #0) == Info: DO phase starts == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_SFTP_QUOTE_INIT (line 2332) == Info: SSH 0x80048f90 state change from SSH_SFTP_QUOTE_INIT to SSH_SFTP_GETINFO (line 868) == Info: SSH 0x80048f90 state change from SSH_SFTP_GETINFO to SSH_SFTP_TRANS_INIT (line 1060) == Info: SSH 0x80048f90 state change from SSH_SFTP_TRANS_INIT to SSH_SFTP_UPLOAD_INIT (line 1080) == Info: SSH 0x80048f90 state change from SSH_SFTP_UPLOAD_INIT to SSH_SFTP_CLOSE (line 1143) == Info: STATE: DO => DOING handle 0x80068340; line 1608 (connection #0) == Info: SFTP DONE done == Info: SSH 0x80048f90 state change from SSH_SFTP_CLOSE to SSH_STOP (line 1651) == Info: DO phase is complete == Info: STATE: DOING => DO_DONE handle 0x80068340; line 1693 (connection #0) == Info: STATE: DO_DONE => PERFORM handle 0x80068340; line 1743 (connection #0) == Info: STATE: PERFORM => DONE handle 0x80068340; line 1933 (connection #0) == Info: multi_done == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_SFTP_CLOSE (line 2390) == Info: SFTP DONE done == Info: SSH 0x80048f90 state change from SSH_SFTP_CLOSE to SSH_STOP (line 1651) == Info: Connection #0 to host XXX left intact Trace from subsequent call:== Info: STATE: INIT => CONNECT handle 0x80068340; line 1356 (connection #-5000) == Info: Added connection 0. The cache now contains 1 members == Info: STATE: CONNECT =>
Re: cURL uploads 0 length file for sftp transfer.
On 2019-09-06 14:17, Yaakov Selkowitz wrote: > On Fri, 2019-09-06 at 19:44 +, ALbert via cygwin wrote: >> We recently upgraded cURL from version:curl 7.50.3 (i686-pc-cygwin) >> libcurl/7.50.3 OpenSSL/1.0.2j zlib/1.2.8 libidn/1.29 libpsl/0.14.0 >> (+libidn/1.29) libssh2/1.7.0 nghttp2/1.14.0 to curl 7.64.1 (i686-pc- >> cygwin) libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.11 brotli/1.0.7 >> libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4) >> libssh/0.8.7/openssl/zlib nghttp2/1.37.0 > > Am I reading this right, curl 7.64.1 with libcurl 7.65.3? Those > versions should match exactly. > >> Since the upgrade, we've seen problems uploading text files to sftp servers. >> Here's the call that we make: >> curl--trace-ascii trace.txt -k sftp://:@/file.txt-T >> file.txt >> >> On initial call, a 0-length file is uploaded, with successful return code. >> On immediate rerun, a full file is uploaded (if we wait few minutes, >> 0-length file will get uploaded). Below are the traces from initial and >> subsequent calls. We also tried using same version curl on a linux box, >> without any issues. We also downloaded latest version of curl from curl for >> Windows, "curl 7.65.3 (i386-pc-win32) libcurl/7.65.3 OpenSSL/1.1.1c >> (Schannel) zlib/1.2.11 brotli/1.0.7 WinIDN libssh2/1.9.0 nghttp2/1.39.2", >> and its also working correctly. >> Unfortunately, the server we're uploading to belongs to a client, and we >> can't see if this is an issue on a server side. However, even if it was, why >> would linux curl and windows curl (from curl.haxx.se) would work correctly? >> Any help or pointers would be appreciated. Thank you. > > Make sure you are using all the latest packages, especially that you > have the latest and matching versions of curl and libcurl, and try > again. FYI Cygwin curl current version produces: $ curl --version curl 7.65.3 (x86_64-pc-cygwin) libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4) libssh/0.8.7/openssl/zlib nghttp2/1.37.0 Release-Date: 2019-07-19 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP TrackMemory UnixSockets -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: cURL uploads 0 length file for sftp transfer.
On Fri, 2019-09-06 at 19:44 +, ALbert via cygwin wrote: > We recently upgraded cURL from version:curl 7.50.3 (i686-pc-cygwin) > libcurl/7.50.3 OpenSSL/1.0.2j zlib/1.2.8 libidn/1.29 libpsl/0.14.0 > (+libidn/1.29) libssh2/1.7.0 nghttp2/1.14.0 to curl 7.64.1 (i686-pc- > cygwin) libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.11 brotli/1.0.7 > libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4) > libssh/0.8.7/openssl/zlib nghttp2/1.37.0 Am I reading this right, curl 7.64.1 with libcurl 7.65.3? Those versions should match exactly. > Since the upgrade, we've seen problems uploading text files to sftp servers. > Here's the call that we make: > curl--trace-ascii trace.txt -k sftp://:@/file.txt-T > file.txt > > On initial call, a 0-length file is uploaded, with successful return code. On > immediate rerun, a full file is uploaded (if we wait few minutes, 0-length > file will get uploaded). Below are the traces from initial and subsequent > calls. We also tried using same version curl on a linux box, without any > issues. We also downloaded latest version of curl from curl for Windows, > "curl 7.65.3 (i386-pc-win32) libcurl/7.65.3 OpenSSL/1.1.1c (Schannel) > zlib/1.2.11 brotli/1.0.7 WinIDN libssh2/1.9.0 nghttp2/1.39.2", and its also > working correctly. > Unfortunately, the server we're uploading to belongs to a client, and we > can't see if this is an issue on a server side. However, even if it was, why > would linux curl and windows curl (from curl.haxx.se) would work correctly? > Any help or pointers would be appreciated. Thank you. Make sure you are using all the latest packages, especially that you have the latest and matching versions of curl and libcurl, and try again. -- Yaakov -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
cURL uploads 0 length file for sftp transfer.
We recently upgraded cURL from version:curl 7.50.3 (i686-pc-cygwin) libcurl/7.50.3 OpenSSL/1.0.2j zlib/1.2.8 libidn/1.29 libpsl/0.14.0 (+libidn/1.29) libssh2/1.7.0 nghttp2/1.14.0tocurl 7.64.1 (i686-pc-cygwin) libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.0.4) libssh/0.8.7/openssl/zlib nghttp2/1.37.0 Since the upgrade, we've seen problems uploading text files to sftp servers. Here's the call that we make: curl--trace-ascii trace.txt -k sftp://:@/file.txt-T file.txt On initial call, a 0-length file is uploaded, with successful return code. On immediate rerun, a full file is uploaded (if we wait few minutes, 0-length file will get uploaded). Below are the traces from initial and subsequent calls. We also tried using same version curl on a linux box, without any issues. We also downloaded latest version of curl from curl for Windows, "curl 7.65.3 (i386-pc-win32) libcurl/7.65.3 OpenSSL/1.1.1c (Schannel) zlib/1.2.11 brotli/1.0.7 WinIDN libssh2/1.9.0 nghttp2/1.39.2", and its also working correctly. Unfortunately, the server we're uploading to belongs to a client, and we can't see if this is an issue on a server side. However, even if it was, why would linux curl and windows curl (from curl.haxx.se) would work correctly? Any help or pointers would be appreciated. Thank you. Here's the trace files: >From initial call:== Info: STATE: INIT => CONNECT handle 0x80068340; line 1356 >(connection #-5000) == Info: Added connection 0. The cache now contains 1 members == Info: STATE: CONNECT => WAITRESOLVE handle 0x80068340; line 1397 (connection #0) == Info: Trying XXX.XX.XX.XX...== Info: TCP_NODELAY set == Info: STATE: WAITRESOLVE => WAITCONNECT handle 0x80068340; line 1476 (connection #0) == Info: Connected to XXX ( XXX.XX.XX.XX) port 22 (#0)== Info: STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x80068340; line 1532 (connection #0) == Info: Marked for [keep alive]: SSH default == Info: User: Opus_efg == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_INIT (line 2115) == Info: SSH 0x80048f90 state change from SSH_INIT to SSH_S_STARTUP (line 582) == Info: STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x80068340; line 1547 (connection #0) == Info: SSH 0x80048f90 state change from SSH_S_STARTUP to SSH_HOSTKEY (line 595) == Info: SSH 0x80048f90 state change from SSH_HOSTKEY to SSH_AUTHLIST (line 605) == Info: SSH 0x80048f90 state change from SSH_AUTHLIST to SSH_AUTH_PKEY_INIT (line 628) == Info: Authentication using SSH public key file == Info: SSH 0x80048f90 state change from SSH_AUTH_PKEY_INIT to SSH_AUTH_KEY_INIT (line 697) == Info: SSH 0x80048f90 state change from SSH_AUTH_KEY_INIT to SSH_AUTH_KEY (line 743) == Info: completed keyboard interactive authentication == Info: SSH 0x80048f90 state change from SSH_AUTH_KEY to SSH_AUTH_DONE (line 761) == Info: Authentication complete == Info: SSH 0x80048f90 state change from SSH_AUTH_DONE to SSH_SFTP_INIT (line 807) == Info: SSH 0x80048f90 state change from SSH_SFTP_INIT to SSH_SFTP_REALPATH (line 833) == Info: SSH CONNECT phase done == Info: SSH 0x80048f90 state change from SSH_SFTP_REALPATH to SSH_STOP (line 850) == Info: STATE: PROTOCONNECT => DO handle 0x80068340; line 1566 (connection #0) == Info: DO phase starts == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_SFTP_QUOTE_INIT (line 2332) == Info: SSH 0x80048f90 state change from SSH_SFTP_QUOTE_INIT to SSH_SFTP_GETINFO (line 868) == Info: SSH 0x80048f90 state change from SSH_SFTP_GETINFO to SSH_SFTP_TRANS_INIT (line 1060) == Info: SSH 0x80048f90 state change from SSH_SFTP_TRANS_INIT to SSH_SFTP_UPLOAD_INIT (line 1080) == Info: SSH 0x80048f90 state change from SSH_SFTP_UPLOAD_INIT to SSH_SFTP_CLOSE (line 1143) == Info: STATE: DO => DOING handle 0x80068340; line 1608 (connection #0) == Info: SFTP DONE done == Info: SSH 0x80048f90 state change from SSH_SFTP_CLOSE to SSH_STOP (line 1651) == Info: DO phase is complete == Info: STATE: DOING => DO_DONE handle 0x80068340; line 1693 (connection #0) == Info: STATE: DO_DONE => PERFORM handle 0x80068340; line 1743 (connection #0) == Info: STATE: PERFORM => DONE handle 0x80068340; line 1933 (connection #0) == Info: multi_done == Info: SSH 0x80048f90 state change from SSH_STOP to SSH_SFTP_CLOSE (line 2390) == Info: SFTP DONE done == Info: SSH 0x80048f90 state change from SSH_SFTP_CLOSE to SSH_STOP (line 1651) == Info: Connection #0 to host XXX left intact Trace from subsequent call:== Info: STATE: INIT => CONNECT handle 0x80068340; line 1356 (connection #-5000) == Info: Added connection 0. The cache now contains 1 members == Info: STATE: CONNECT => WAITRESOLVE handle 0x80068340; line 1397 (connection #0) == Info: Trying XXX.XX.XX.XX...== Info: TCP_NODELAY set == Info: STATE: WAITRESOLVE => WAITCONNECT handle 0x80068340; line 1476 (connection #0) == Info: Connected to XXX (XXX.XX.XX.XX) port 22 (#0) == Info: STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x80068340;
