Re: error: git-remote-https died of signal 13

[Greg M]

On Thu, Apr 24, 2014 at 12:15 AM, Jeff King p...@peff.net wrote:
 I suspect the curl patch below may fix it:

 diff --git a/lib/multi.c b/lib/multi.c
 index bc93264..72e4825 100644
 --- a/lib/multi.c
 +++ b/lib/multi.c
 @@ -1804,10 +1804,13 @@ static void close_all_connections(struct Curl_multi 
 *multi)

conn = Curl_conncache_find_first_connection(multi-conn_cache);
while(conn) {
 +SIGPIPE_VARIABLE(pipe_st);
  conn-data = multi-closure_handle;

 +sigpipe_ignore(conn-data, pipe_st);
  /* This will remove the connection from the cache */
  (void)Curl_disconnect(conn, FALSE);
 +sigpipe_restore(pipe_st);

  conn = Curl_conncache_find_first_connection(multi-conn_cache);
}

The patch fixes the problem,

Greg
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Daniel Stenberg]

On Thu, 24 Apr 2014, Jeff King wrote:

Thanks, that's very helpful. I wasn't able to reproduce your problem 
locally, but I suspect the curl patch below may fix it:


...

Daniel, I think the similar fix to curl_multi_cleanup in commit a900d45 
missed this code path, and we need something like the above patch. I know 
you were trying to keep the SIGPIPE mess at the entrance-points to the 
library, and this works against that. But we need a SessionHandle to pass to 
sigpipe_ignore to look at its no_signal flag, and of course in the case of 
multi we may have several such handles. If there's a similar flag we can 
check on the multi handle, we could just cover all of curl_multi_cleanup 
with a single ignore/reset pair.


Thanks a lot for this! I've taken this issue over to the libcurl mailing list 
and we'll work out the best way to address it over there... At least we know 
the patch works as intended.


--

 / daniel.haxx.se
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Jeff King]

On Sun, Apr 20, 2014 at 08:42:15PM -0400, Greg M wrote:

 Using git version 1.9.2 I am getting this error:
 
 [normal@laptop tmp]$ git clone https://github.com/mozilla/rust.git
 Cloning into 'rust'...
 remote: Reusing existing pack: 296648, done.
 remote: Counting objects: 80, done.
 remote: Compressing objects: 100% (77/77), done.
 remote: Total 296728 (delta 22), reused 9 (delta 3)
 Receiving objects: 100% (296728/296728), 110.68 MiB | 190.00 KiB/s, done.
 Resolving deltas: 100% (238828/238828), done.
 Checking connectivity... done.
 error: git-remote-https died of signal 13

Thanks for a thorough bug report. I looked through your strace output,
and this really does look like another case of OpenSSL getting SIGPIPE
while closing the connection.

It's odd, though, as your curl version has my patches, along with
similar extra fixes in 854aca5 (multi: ignore sigpipe internally,
2014-02-17). But I guess there may be a code path that needs similar
treatment.

The easiest way to find it is probably to attach a debugger to the
running git-remote-https, and get a backtrace when it dies from SIGPIPE.
You'll probably want to install your system's debug packages for curl,
too.

 I have curl version 7.36 though, in case some of the other output matters:
 
 [normal@laptop tmp]$ curl --version
 curl 7.36.0 (x86_64-unknown-linux-gnu) libcurl/7.36.0 OpenSSL/1.0.1g
 zlib/1.2.8 libssh2/1.4.3

Another possibility is that your curl binary is up-to-date, but you are
linking against an older version of libcurl that does not have the
SIGPIPE workarounds.

I'm not sure of the best way to check that, but a hacky way under Linux
is:

  $ ldd $(git --exec-path)/git-remote-https | grep libcurl
  libcurl.so.4 = /usr/lib/x86_64-linux-gnu/libcurl.so.4
  $ strings  /usr/lib/x86_64-linux-gnu/libcurl.so.4 | grep '7\.'
  CLIENT libcurl 7.36.0

-Peff
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Greg M]

On Wed, Apr 23, 2014 at 2:59 AM, Jeff King p...@peff.net wrote:
 On Sun, Apr 20, 2014 at 08:42:15PM -0400, Greg M wrote:

 Using git version 1.9.2 I am getting this error:

 [normal@laptop tmp]$ git clone https://github.com/mozilla/rust.git
 Cloning into 'rust'...
 remote: Reusing existing pack: 296648, done.
 remote: Counting objects: 80, done.
 remote: Compressing objects: 100% (77/77), done.
 remote: Total 296728 (delta 22), reused 9 (delta 3)
 Receiving objects: 100% (296728/296728), 110.68 MiB | 190.00 KiB/s, done.
 Resolving deltas: 100% (238828/238828), done.
 Checking connectivity... done.
 error: git-remote-https died of signal 13

 Thanks for a thorough bug report. I looked through your strace output,
 and this really does look like another case of OpenSSL getting SIGPIPE
 while closing the connection.

 It's odd, though, as your curl version has my patches, along with
 similar extra fixes in 854aca5 (multi: ignore sigpipe internally,
 2014-02-17). But I guess there may be a code path that needs similar
 treatment.

 The easiest way to find it is probably to attach a debugger to the
 running git-remote-https, and get a backtrace when it dies from SIGPIPE.
 You'll probably want to install your system's debug packages for curl,
 too.


The backtrace:

Program received signal SIGPIPE, Broken pipe.
0x7fdcfd511a2d in write () from /usr/lib/libpthread.so.0
(gdb) bt
#0  0x7fdcfd511a2d in write () from /usr/lib/libpthread.so.0
#1  0x7fdcfd81a0f6 in sock_write () from /usr/lib/libcrypto.so.1.0.0
#2  0x7fdcfd817edb in BIO_write () from /usr/lib/libcrypto.so.1.0.0
#3  0x7fdcfc662902 in ssl3_write_pending () from /usr/lib/libssl.so.1.0.0
#4  0x7fdcfc664b77 in ssl3_dispatch_alert () from /usr/lib/libssl.so.1.0.0
#5  0x7fdcfc660822 in ssl3_shutdown () from /usr/lib/libssl.so.1.0.0
#6  0x7fdcfd2e944e in Curl_ossl_close () from /usr/lib/libcurl.so.4
#7  0x7fdcfd2b6459 in Curl_disconnect () from /usr/lib/libcurl.so.4
#8  0x7fdcfd2c8131 in curl_multi_cleanup () from /usr/lib/libcurl.so.4
#9  0x0040764b in ?? ()
#10 0x00404e4d in ?? ()
#11 0x7fdcfcf0fb05 in __libc_start_main () from /usr/lib/libc.so.6
#12 0x0040552e in ?? ()

 Another possibility is that your curl binary is up-to-date, but you are
 linking against an older version of libcurl that does not have the
 SIGPIPE workarounds.

 I'm not sure of the best way to check that, but a hacky way under Linux
 is:

   $ ldd $(git --exec-path)/git-remote-https | grep libcurl
   libcurl.so.4 = /usr/lib/x86_64-linux-gnu/libcurl.so.4
   $ strings  /usr/lib/x86_64-linux-gnu/libcurl.so.4 | grep '7\.'
   CLIENT libcurl 7.36.0

 -Peff

Seems to actually be that version of libcurl:

[normal@laptop tmp]$ ldd $(git --exec-path)/git-remote-https | grep libcurl
libcurl.so.4 = /usr/lib/libcurl.so.4 (0x7ff1cce5e000)
[normal@laptop tmp]$ strings /usr/lib/libcurl.so.4 | grep '7\.'
7.36f
CLIENT libcurl 7.36.0
CLIENT libcurl 7.36.0
CLIENT libcurl 7.36.0
7.36.0
[normal@laptop tmp]$

Also I don't think I have any other versions to link against:

[normal@laptop tmp]$ ls -l /usr/lib/libcurl*
lrwxrwxrwx 1 root root 16 Mar 26 10:12 /usr/lib/libcurl.so -
libcurl.so.4.3.0
lrwxrwxrwx 1 root root 16 Mar 26 10:12 /usr/lib/libcurl.so.4 -
libcurl.so.4.3.0
-rwxr-xr-x 1 root root 443488 Mar 26 10:12 /usr/lib/libcurl.so.4.3.0

Greg
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Jeff King]

On Wed, Apr 23, 2014 at 07:49:09AM -0400, Greg M wrote:

  The easiest way to find it is probably to attach a debugger to the
  running git-remote-https, and get a backtrace when it dies from SIGPIPE.
  You'll probably want to install your system's debug packages for curl,
  too.
 
 
 The backtrace:
 
 Program received signal SIGPIPE, Broken pipe.
 0x7fdcfd511a2d in write () from /usr/lib/libpthread.so.0
 (gdb) bt
 #0  0x7fdcfd511a2d in write () from /usr/lib/libpthread.so.0
 #1  0x7fdcfd81a0f6 in sock_write () from /usr/lib/libcrypto.so.1.0.0
 #2  0x7fdcfd817edb in BIO_write () from /usr/lib/libcrypto.so.1.0.0
 #3  0x7fdcfc662902 in ssl3_write_pending () from /usr/lib/libssl.so.1.0.0
 #4  0x7fdcfc664b77 in ssl3_dispatch_alert () from /usr/lib/libssl.so.1.0.0
 #5  0x7fdcfc660822 in ssl3_shutdown () from /usr/lib/libssl.so.1.0.0
 #6  0x7fdcfd2e944e in Curl_ossl_close () from /usr/lib/libcurl.so.4
 #7  0x7fdcfd2b6459 in Curl_disconnect () from /usr/lib/libcurl.so.4
 #8  0x7fdcfd2c8131 in curl_multi_cleanup () from /usr/lib/libcurl.so.4
 #9  0x0040764b in ?? ()
 #10 0x00404e4d in ?? ()
 #11 0x7fdcfcf0fb05 in __libc_start_main () from /usr/lib/libc.so.6
 #12 0x0040552e in ?? ()

Thanks, that's very helpful. I wasn't able to reproduce your problem
locally, but I suspect the curl patch below may fix it:

diff --git a/lib/multi.c b/lib/multi.c
index bc93264..72e4825 100644
--- a/lib/multi.c
+++ b/lib/multi.c
@@ -1804,10 +1804,13 @@ static void close_all_connections(struct Curl_multi 
*multi)
 
   conn = Curl_conncache_find_first_connection(multi-conn_cache);
   while(conn) {
+SIGPIPE_VARIABLE(pipe_st);
 conn-data = multi-closure_handle;
 
+sigpipe_ignore(conn-data, pipe_st);
 /* This will remove the connection from the cache */
 (void)Curl_disconnect(conn, FALSE);
+sigpipe_restore(pipe_st);
 
 conn = Curl_conncache_find_first_connection(multi-conn_cache);
   }

Let me know if you need any pointers on getting curl built or tested
with git.

Daniel, I think the similar fix to curl_multi_cleanup in commit a900d45
missed this code path, and we need something like the above patch. I
know you were trying to keep the SIGPIPE mess at the entrance-points to
the library, and this works against that. But we need a SessionHandle to
pass to sigpipe_ignore to look at its no_signal flag, and of course in
the case of multi we may have several such handles. If there's a similar
flag we can check on the multi handle, we could just cover all of
curl_multi_cleanup with a single ignore/reset pair.

-Peff
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

error: git-remote-https died of signal 13

[Greg M]

Hi,

Using git version 1.9.2 I am getting this error:

[normal@laptop tmp]$ git clone https://github.com/mozilla/rust.git
Cloning into 'rust'...
remote: Reusing existing pack: 296648, done.
remote: Counting objects: 80, done.
remote: Compressing objects: 100% (77/77), done.
remote: Total 296728 (delta 22), reused 9 (delta 3)
Receiving objects: 100% (296728/296728), 110.68 MiB | 190.00 KiB/s, done.
Resolving deltas: 100% (238828/238828), done.
Checking connectivity... done.
error: git-remote-https died of signal 13

The repository appears to be cloned fine, I can clone other repositories
without error.

I appear to have the exact same symptoms as Stefan in November 2013 as
archived here:
http://thread.gmane.org/gmane.comp.version-control.git/238242/focus=238311.
In this case the cause was a curl error that was fixed in version curl
version 7.34...

I have curl version 7.36 though, in case some of the other output matters:

[normal@laptop tmp]$ curl --version
curl 7.36.0 (x86_64-unknown-linux-gnu) libcurl/7.36.0 OpenSSL/1.0.1g
zlib/1.2.8 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp
scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IPv6 Largefile NTLM NTLM_WB SSL libz
TLS-SRP

The curl command to reproduce the since fixed bug (curl --limit-rate 250k
-L http://github.com/raspberrypi/firmware/archive/4ade27942e.tar.gz 
/dev/null) runs without error and exits with a value of 0.

The first response to Stefan's bug report requested the output of the
command with GIT_TRANSPORT_HELPER_DEBUG=1 and strace -f, so I've included
that:

GIT_TRANSPORT...:
https://drive.google.com/file/d/0BxdiDTQp3MYXQWJJaVdydUstbWs/edit?usp=sharing
strace (33M download, 330M uncompressed):
https://drive.google.com/file/d/0BxdiDTQp3MYXb1VKWWtFUDdvbjQ/edit?usp=sharing

Thanks,
Greg
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Jeff King]

On Mon, Nov 25, 2013 at 08:20:18AM +0100, Daniel Stenberg wrote:

 Daniel, does the call to Curl_disconnect need to be wrapped with
 sigpipe_ignore/reset, similar to 7d80ed64e435155?
 
 Yes. It very much looks like that. The SSL closing is what was the
 problem I had to adress.
 
 But I then decided that if a 3rd library has one way to generate
 SIGPIPE it may very well have another in a separate spot so I decided
 to do the wrap at the top level immediately in the entry point when
 getting called by the application. Following that, the SIGPIPE
 ignore/restore should rather be made in curl_multi_cleanup.

Unfortunately, we need an actual SessionHandle to know whether it is OK
to reset signals at all. There may be a more elegant way of checking
that, but here's the patch series I came up with. It does turn off
SIGPIPE for the specific case I'm seeing (again, I'm having trouble
actually getting EPIPE in the first place, but from Stefan's strace, I
think this would fix his problem).

  [1/2]: factor out sigpipe_reset from easy.c
  [2/2]: ignore SIGPIPE during curl_multi_cleanup

-Peff
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Jeff King]

On Mon, Nov 25, 2013 at 09:32:13AM -0500, Jeff King wrote:

  But I then decided that if a 3rd library has one way to generate
  SIGPIPE it may very well have another in a separate spot so I decided
  to do the wrap at the top level immediately in the entry point when
  getting called by the application. Following that, the SIGPIPE
  ignore/restore should rather be made in curl_multi_cleanup.
 
 Unfortunately, we need an actual SessionHandle to know whether it is OK
 to reset signals at all. There may be a more elegant way of checking
 that, but here's the patch series I came up with.

Scratch that.  I had originally written something like:

  if (conn-data)
sigpipe_ignore(conn-data, pipe);
  Curl_disconnect(conn, ...);
  sigpipe_restore(pipe);

but while sending it out, I realized that the data we attach to each
connection when we close it is just the multi-closure_handle. So I was
able to hoist the check out to curl_multi_cleanup (and that's what I
just sent).

-Peff
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Stefan Beller]

On 24.11.2013 07:54, Jeff King wrote:
 On Sat, Nov 23, 2013 at 05:36:36PM +0100, Stefan Beller wrote:
 
 sb@sb:/tmp$ git clone https://github.com/Bertram25/ValyriaTear.git
 Cloning into 'ValyriaTear'...
 remote: Counting objects: 21346, done.
 remote: Compressing objects: 100% (6307/6307), done.
 remote: Total 21346 (delta 16463), reused 19820 (delta 15000)
 Receiving objects: 100% (21346/21346), 176.39 MiB | 445.00 KiB/s, done.
 Resolving deltas: 100% (16463/16463), done.
 Checking connectivity... done.
 error: git-remote-https died of signal 13

 However the repository seems to be cloned fine.
 I can clone different repos from github and they are working fine
 without the error.
 Would that be an issue on my side or on githubs side?
 
 Almost certainly on your side. 13 is SIGPIPE, so git-remote-https is
 trying to write something but the other side of the pipe has hung up.
 This might be a race condition in the transport-helper protocol, where
 we've had similar problems before.
 
 It doesn't reproduce here for me. Can you reproduce it consistently on
 this repo? I would not be at all surprised if it is intermittent.


I tested 3 times before sending the mail and I always got the error.
Testing now doesn't always trigger this problem. So yeah it's kind of 
intermittent.
 
 If you can reproduce, it would be interesting to see the output with
 GIT_TRANSPORT_HELPER_DEBUG=1, or even with strace -f. That could at
 least tell us what it was trying to write (and to where) when it got the
 SIGPIPE.
 
 -Peff
 

Here is the output of 
sb@sb:/tmp$ GIT_TRANSPORT_HELPER_DEBUG=1 git clone 
https://github.com/Bertram25/ValyriaTear.git tmp
Cloning into 'tmp'...
Debug: Remote helper: - capabilities
Debug: Remote helper: Waiting...
Debug: Remote helper: - fetch
Debug: Got cap fetch
Debug: Remote helper: Waiting...
Debug: Remote helper: - option
Debug: Got cap option
Debug: Remote helper: Waiting...
Debug: Remote helper: - push
Debug: Got cap push
Debug: Remote helper: Waiting...
Debug: Remote helper: - check-connectivity
Debug: Got cap check-connectivity
Debug: Remote helper: Waiting...
Debug: Remote helper: - 
Debug: Capabilities complete.
Debug: Remote helper: Waiting...
Debug: Remote helper: - 064275ea9c91688442ab215b5b641ac93fdd77b0 HEAD
Debug: Remote helper: Waiting...
Debug: Remote helper: - a32429792c9b393732bd026d42374f226c928559 refs/heads/0.6
Debug: Remote helper: Waiting...
Debug: Remote helper: - 6153fdaf34d9d3eec5fc1541b0373db6ef0eac25 
refs/heads/HEI-Release
Debug: Remote helper: Waiting...
Debug: Remote helper: - 9d452017c18389120c662bfe5a3379a0c15458e7 
refs/heads/HeroOfAllacrost
Debug: Remote helper: Waiting...
Debug: Remote helper: - 064275ea9c91688442ab215b5b641ac93fdd77b0 
refs/heads/master
Debug: Remote helper: Waiting...
Debug: Remote helper: - f5a3727f88d5c48f3c28af85bd32567772314c1a 
refs/heads/sdl2
Debug: Remote helper: Waiting...
Debug: Remote helper: - aa2cc4a7961e5c85db8c971f8ca15febeea84ac9 
refs/pull/101/head
Debug: Remote helper: Waiting...
Debug: Remote helper: - b707dc07da480829b46b31b114fa3335ac10066f 
refs/pull/101/merge
Debug: Remote helper: Waiting...
Debug: Remote helper: - 995d48764fd0485e1fd2286768926d33f436b983 
refs/pull/103/head
Debug: Remote helper: Waiting...
Debug: Remote helper: - 40c6020443fabf86b9516567caf4aa5b4478022d 
refs/pull/103/merge
Debug: Remote helper: Waiting...
Debug: Remote helper: - 0fc5e0abd6e5070df90b155a54c80ec07aabb1f0 
refs/pull/111/head
Debug: Remote helper: Waiting...
Debug: Remote helper: - dd855fe5f09f9984dad933b9512305b9ab539c55 
refs/pull/111/merge
Debug: Remote helper: Waiting...
Debug: Remote helper: - de0347d77025ad8b87da616c487c650b28834182 
refs/pull/114/head
Debug: Remote helper: Waiting...
Debug: Remote helper: - 224b18b50c2a0a58c6566972b7d85a5451f5c3aa 
refs/pull/114/merge
Debug: Remote helper: Waiting...
Debug: Remote helper: - fae33cbba4cf3c0a786f23ae6ef1808c77e4d79f 
refs/pull/115/head
Debug: Remote helper: Waiting...
Debug: Remote helper: - 1659dffb9828946b77fc41ef175a91f94d908e55 
refs/pull/115/merge
Debug: Remote helper: Waiting...
Debug: Remote helper: - ee99c3b5c2e31482c89826c7821a8feadda0ec56 
refs/pull/116/head
Debug: Remote helper: Waiting...
Debug: Remote helper: - ab9e412654cc8a3f50fcf63e73aa6232e35bd7a9 
refs/pull/116/merge
Debug: Remote helper: Waiting...
Debug: Remote helper: - a04786140b697f3d732caae99984d94c66d8da57 
refs/pull/117/head
Debug: Remote helper: Waiting...
Debug: Remote helper: - b07e621e4bce831daf2881eb128b1ebef2b83414 
refs/pull/117/merge
Debug: Remote helper: Waiting...
Debug: Remote helper: - c303913334cadcaefecb7ef8df51b9437a0d9d65 
refs/pull/118/head
Debug: Remote helper: Waiting...
Debug: Remote helper: - fd074efa3286be31b92c6380a85ce15799f3ce7d 
refs/pull/118/merge
Debug: Remote helper: Waiting...
Debug: Remote helper: - 5eb3c686e430c8337652e56313a23a394482b552 
refs/pull/119/head
Debug: Remote helper: Waiting...
Debug: Remote helper: - a69fdef2ab67447024133a84ce3835519809ef3a 
refs/pull/119/merge
Debug

Re: error: git-remote-https died of signal 13

[Jeff King]

On Sun, Nov 24, 2013 at 01:54:34PM +0100, Stefan Beller wrote:

 Here is the output of 
 sb@sb:/tmp$ GIT_TRANSPORT_HELPER_DEBUG=1 git clone 
 https://github.com/Bertram25/ValyriaTear.git tmp

Thanks. I think I see what is going on.

We finish the helper conversation here:

 Checking connectivity... done.
 Debug: Disconnecting.
 error: git-remote-https died of signal 13
 sb@sb:/tmp$ 

which means that remote-https is trying to exit, and is cleaning up any
curl connections. The actual SIGPIPE in the strace is here:

[pid 28319] write(3, \25\3\2\0...[binary goo]..., 27) = -1 EPIPE (Broken pipe)

and if you walk backwards, fd 3 is:

  [pid 28319] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
  ...
  [pid 28319] connect(3, {sa_family=AF_INET,
  sin_port=htons(443),
  sin_addr=inet_addr(192.30.252.131)}, 16
  ) = -1 EINPROGRESS (Operation now in progress)

So it's sending binary junk to the https socket while trying to exit,
which makes me guess that it's something to do with terminating the SSL
session, but the server has already hung up. Which would make it a curl
problem.

Googling curl sigpipe seems to come up with a report of this exact
case:

  http://curl.haxx.se/mail/archive-2013-01/0003.html

with a bug opened here:

  http://sourceforge.net/p/curl/bugs/1180/

Looks like the fix went into curl 7.32.0. I have 7.33.0, which seems
fine. Can you confirm that your libcurl is a bit older?

-Peff
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Stefan Beller]

On 24.11.2013 14:33, Jeff King wrote:
 On Sun, Nov 24, 2013 at 01:54:34PM +0100, Stefan Beller wrote:
 
 Here is the output of 
 sb@sb:/tmp$ GIT_TRANSPORT_HELPER_DEBUG=1 git clone 
 https://github.com/Bertram25/ValyriaTear.git tmp
 
 Thanks. I think I see what is going on.
 
 We finish the helper conversation here:
 
 Checking connectivity... done.
 Debug: Disconnecting.
 error: git-remote-https died of signal 13
 sb@sb:/tmp$ 
 
 which means that remote-https is trying to exit, and is cleaning up any
 curl connections. The actual SIGPIPE in the strace is here:
 
 [pid 28319] write(3, \25\3\2\0...[binary goo]..., 27) = -1 EPIPE (Broken 
 pipe)
 
 and if you walk backwards, fd 3 is:
 
   [pid 28319] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
   ...
   [pid 28319] connect(3, {sa_family=AF_INET,
   sin_port=htons(443),
   sin_addr=inet_addr(192.30.252.131)}, 16
   ) = -1 EINPROGRESS (Operation now in progress)
 
 So it's sending binary junk to the https socket while trying to exit,
 which makes me guess that it's something to do with terminating the SSL
 session, but the server has already hung up. Which would make it a curl
 problem.
 
 Googling curl sigpipe seems to come up with a report of this exact
 case:
 
   http://curl.haxx.se/mail/archive-2013-01/0003.html

I cannot reproduce the error using the curl command from that site.
curl returns with 0.

 
 with a bug opened here:
 
   http://sourceforge.net/p/curl/bugs/1180/
 
 Looks like the fix went into curl 7.32.0. I have 7.33.0, which seems
 fine. Can you confirm that your libcurl is a bit older?
 

dpkg -l |grep curl
ii  curl  7.32.0-1ubuntu1   
 amd64command line tool for transferring data with URL syntax
ii  libcurl3:amd647.32.0-1ubuntu1   
 amd64easy-to-use client-side URL transfer library (OpenSSL 
flavour)
ii  libcurl3-gnutls:amd64 7.32.0-1ubuntu1   
 amd64easy-to-use client-side URL transfer library (GnuTLS 
flavour)
ii  libcurl4-openssl-dev  7.32.0-1ubuntu1   
 amd64development files and documentation for libcurl (OpenSSL 
flavour)
ii  python-pycurl 7.19.0-5ubuntu8   
 amd64Python bindings to libcurl

Thanks,
Stefan
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Jeff King]

[+cc Daniel, who worked on the curl fix]

On Sun, Nov 24, 2013 at 04:01:43PM +0100, Stefan Beller wrote:

 On 24.11.2013 14:33, Jeff King wrote:
  On Sun, Nov 24, 2013 at 01:54:34PM +0100, Stefan Beller wrote:
  
  Here is the output of 
  sb@sb:/tmp$ GIT_TRANSPORT_HELPER_DEBUG=1 git clone 
  https://github.com/Bertram25/ValyriaTear.git tmp
  
  Thanks. I think I see what is going on.
  
  We finish the helper conversation here:
  
  Checking connectivity... done.
  Debug: Disconnecting.
  error: git-remote-https died of signal 13
  sb@sb:/tmp$ 
  
  which means that remote-https is trying to exit, and is cleaning up any
  curl connections. The actual SIGPIPE in the strace is here:
  
  [pid 28319] write(3, \25\3\2\0...[binary goo]..., 27) = -1 EPIPE (Broken 
  pipe)
  
  and if you walk backwards, fd 3 is:
  
[pid 28319] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
...
[pid 28319] connect(3, {sa_family=AF_INET,
sin_port=htons(443),
sin_addr=inet_addr(192.30.252.131)}, 16
) = -1 EINPROGRESS (Operation now in progress)
  
  So it's sending binary junk to the https socket while trying to exit,
  which makes me guess that it's something to do with terminating the SSL
  session, but the server has already hung up. Which would make it a curl
  problem.
  
  Googling curl sigpipe seems to come up with a report of this exact
  case:
  
http://curl.haxx.se/mail/archive-2013-01/0003.html
 
 I cannot reproduce the error using the curl command from that site.
 curl returns with 0.
 
  
  with a bug opened here:
  
http://sourceforge.net/p/curl/bugs/1180/
  
  Looks like the fix went into curl 7.32.0. I have 7.33.0, which seems
  fine. Can you confirm that your libcurl is a bit older?
  
 
 dpkg -l |grep curl
 ii  curl  7.32.0-1ubuntu1 
amd64command line tool for transferring data with URL 
 syntax
 ii  libcurl3:amd647.32.0-1ubuntu1 
amd64easy-to-use client-side URL transfer library (OpenSSL 
 flavour)
 ii  libcurl3-gnutls:amd64 7.32.0-1ubuntu1 
amd64easy-to-use client-side URL transfer library (GnuTLS 
 flavour)
 ii  libcurl4-openssl-dev  7.32.0-1ubuntu1 
amd64development files and documentation for libcurl 
 (OpenSSL flavour)
 ii  python-pycurl 7.19.0-5ubuntu8 
amd64Python bindings to libcurl

Hmm. The fix in curl's 7d80ed64e435155 seems to involve strategically
placed calls to ignore SIGPIPE. I wonder if there is another spot that
needs similar treatment. It looks like curl_easy_cleanup is covered,
though, and that's where I would expect problem to come.

It would be interesting to see a backtrace from remote-curl when we get
the SIGPIPE. Doing so would be slightly tricky; instrumenting with the
patch below may be enough.

Another thought is that the curl fix seems to only kick in when built
with openssl support.  I'm not sure I understand how ubuntu's packaging
of curl uses gnutls versus openssl for the shared library. That may be
related.

-Peff

---
diff --git a/http.c b/http.c
index bcf54aa..ac709cc 100644
--- a/http.c
+++ b/http.c
@@ -473,13 +473,17 @@ void http_cleanup(void)
 {
struct active_request_slot *slot = active_queue_head;
 
+   warning(in http_cleanup);
while (slot != NULL) {
struct active_request_slot *next = slot-next;
if (slot-curl != NULL) {
 #ifdef USE_CURL_MULTI
+   warning(calling curl_multi_remove_handle);
curl_multi_remove_handle(curlm, slot-curl);
 #endif
+   warning(calling curl_easy_cleanup on slot);
curl_easy_cleanup(slot-curl);
+   warning(curl_easy_cleanup done);
}
free(slot);
slot = next;
@@ -487,13 +491,19 @@ void http_cleanup(void)
active_queue_head = NULL;
 
 #ifndef NO_CURL_EASY_DUPHANDLE
+   warning(calling curl_easy_cleanup on default);
curl_easy_cleanup(curl_default);
+   warning(curl_easy_cleanup done);
 #endif
 
 #ifdef USE_CURL_MULTI
+   warning(calling curl_multi_cleanup);
curl_multi_cleanup(curlm);
+   warning(curl_multi_cleanup done);
 #endif
+   warning(calling curl_global_cleanup);
curl_global_cleanup();
+   warning(curl_global_cleanup done);
 
curl_slist_free_all(pragma_header);
pragma_header = NULL;
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Stefan Beller]

On 24.11.2013 16:54, Jeff King wrote:
 Hmm. The fix in curl's 7d80ed64e435155 seems to involve strategically
 placed calls to ignore SIGPIPE. I wonder if there is another spot that
 needs similar treatment. It looks like curl_easy_cleanup is covered,
 though, and that's where I would expect problem to come.
 
 It would be interesting to see a backtrace from remote-curl when we get
 the SIGPIPE. Doing so would be slightly tricky; instrumenting with the
 patch below may be enough.

Ok I'll test that now.

 
 Another thought is that the curl fix seems to only kick in when built
 with openssl support.  I'm not sure I understand how ubuntu's packaging
 of curl uses gnutls versus openssl for the shared library. That may be
 related.
 

A better information would be the --version from curl then:
curl --version
curl 7.32.0 (x86_64-pc-linux-gnu) libcurl/7.32.0 OpenSSL/1.0.1e zlib/1.2.8 
libidn/1.28 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 
pop3s rtmp rtsp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz 
TLS-SRP

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Stefan Beller]

On 24.11.2013 16:54, Jeff King wrote:
 [+cc Daniel, who worked on the curl fix]
 
 On Sun, Nov 24, 2013 at 04:01:43PM +0100, Stefan Beller wrote:
 
 On 24.11.2013 14:33, Jeff King wrote:
 On Sun, Nov 24, 2013 at 01:54:34PM +0100, Stefan Beller wrote:

 Here is the output of 
 sb@sb:/tmp$ GIT_TRANSPORT_HELPER_DEBUG=1 git clone 
 https://github.com/Bertram25/ValyriaTear.git tmp

 Thanks. I think I see what is going on.

 We finish the helper conversation here:

 Checking connectivity... done.
 Debug: Disconnecting.
 error: git-remote-https died of signal 13
 sb@sb:/tmp$ 

 which means that remote-https is trying to exit, and is cleaning up any
 curl connections. The actual SIGPIPE in the strace is here:

 [pid 28319] write(3, \25\3\2\0...[binary goo]..., 27) = -1 EPIPE (Broken 
 pipe)

 and if you walk backwards, fd 3 is:

   [pid 28319] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
   ...
   [pid 28319] connect(3, {sa_family=AF_INET,
   sin_port=htons(443),
   sin_addr=inet_addr(192.30.252.131)}, 16
   ) = -1 EINPROGRESS (Operation now in progress)

 So it's sending binary junk to the https socket while trying to exit,
 which makes me guess that it's something to do with terminating the SSL
 session, but the server has already hung up. Which would make it a curl
 problem.

 Googling curl sigpipe seems to come up with a report of this exact
 case:

   http://curl.haxx.se/mail/archive-2013-01/0003.html

 I cannot reproduce the error using the curl command from that site.
 curl returns with 0.


 with a bug opened here:

   http://sourceforge.net/p/curl/bugs/1180/

 Looks like the fix went into curl 7.32.0. I have 7.33.0, which seems
 fine. Can you confirm that your libcurl is a bit older?


 dpkg -l |grep curl
 ii  curl  7.32.0-1ubuntu1
 amd64command line tool for transferring data with URL 
 syntax
 ii  libcurl3:amd647.32.0-1ubuntu1
 amd64easy-to-use client-side URL transfer library 
 (OpenSSL flavour)
 ii  libcurl3-gnutls:amd64 7.32.0-1ubuntu1
 amd64easy-to-use client-side URL transfer library 
 (GnuTLS flavour)
 ii  libcurl4-openssl-dev  7.32.0-1ubuntu1
 amd64development files and documentation for libcurl 
 (OpenSSL flavour)
 ii  python-pycurl 7.19.0-5ubuntu8
 amd64Python bindings to libcurl
 
 Hmm. The fix in curl's 7d80ed64e435155 seems to involve strategically
 placed calls to ignore SIGPIPE. I wonder if there is another spot that
 needs similar treatment. It looks like curl_easy_cleanup is covered,
 though, and that's where I would expect problem to come.
 
 It would be interesting to see a backtrace from remote-curl when we get
 the SIGPIPE. Doing so would be slightly tricky; instrumenting with the
 patch below may be enough.
 

GIT_TRANSPORT_HELPER_DEBUG=1 git clone 
https://github.com/Bertram25/ValyriaTear.git
now ends with:

Debug: Remote helper: Waiting...
remote: Counting objects: 21354, done.
remote: Compressing objects: 100% (6249/6249), done.
remote: Total 21354 (delta 16466), reused 19888 (delta 15066)
Receiving objects: 100% (21354/21354), 176.42 MiB | 1.22 MiB/s, done.
Resolving deltas: 100% (16466/16466), done.
Debug: Remote helper: - lock 
/tmp/ValyriaTear/.git/objects/pack/pack-b6f360ab28b5078a9aefafe1c4144e6c7782c317.keep
Debug: Remote helper: Waiting...
Debug: Remote helper: - connectivity-ok
Debug: Remote helper: Waiting...
Debug: Remote helper: - 
Checking connectivity... done.
Debug: Disconnecting.
warning: in http_cleanup
warning: calling curl_multi_remove_handle
warning: calling curl_easy_cleanup on slot
warning: curl_easy_cleanup done
warning: calling curl_easy_cleanup on default
warning: curl_easy_cleanup done
warning: calling curl_multi_cleanup
error: git-remote-https died of signal 13

Thanks,
Stefan

--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Daniel Stenberg]

On Sun, 24 Nov 2013, Jeff King wrote:

Hmm. The fix in curl's 7d80ed64e435155 seems to involve strategically placed 
calls to ignore SIGPIPE. I wonder if there is another spot that needs 
similar treatment. It looks like curl_easy_cleanup is covered, though, and 
that's where I would expect problem to come.


Sounds like a plausible reason.

It would be interesting to see a backtrace from remote-curl when we get the 
SIGPIPE. Doing so would be slightly tricky; instrumenting with the patch 
below may be enough.


Another thought is that the curl fix seems to only kick in when built with 
openssl support.  I'm not sure I understand how ubuntu's packaging of curl 
uses gnutls versus openssl for the shared library. That may be related.


I'm only aware of a SIGPIPE problem with openssl that can make it write to the 
socket in some situations when the remote end is no longer there - something 
we can't prevent it from doing.


I *believe* the problem doesn't exist in the similar way when built to use 
gnutls, but I may of course be wrong.


--

 / daniel.haxx.se
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[brian m. carlson]

On Sun, Nov 24, 2013 at 10:54:39AM -0500, Jeff King wrote:
 Another thought is that the curl fix seems to only kick in when built
 with openssl support.  I'm not sure I understand how ubuntu's packaging
 of curl uses gnutls versus openssl for the shared library. That may be
 related.

Debian (and presumably Ubuntu) build the curl source three times: once
each for OpenSSL, GnuTLS, and NSS.  Each shared library is named
differently (libcurl-openssl.so.3, etc.) and in its own package
(libcurl3-openssl).  The corresponding -dev package for each version
sets up the symlinks and install headers to point to the proper version,
so you always compile and link as you expect.

The reason for this is that Debian cannot distribute GPLv2-only programs
(like git) linked against OpenSSL, so GnuTLS becomes necessary.  On
Debian and Ubuntu, git is by default linked against libcurl3-gnutls.

-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187


signature.asc
Description: Digital signature

Re: error: git-remote-https died of signal 13

[Jeff King]

On Sun, Nov 24, 2013 at 05:32:34PM +0100, Stefan Beller wrote:

 GIT_TRANSPORT_HELPER_DEBUG=1 git clone 
 https://github.com/Bertram25/ValyriaTear.git
 now ends with:
 
 Debug: Remote helper: Waiting...
 remote: Counting objects: 21354, done.
 remote: Compressing objects: 100% (6249/6249), done.
 remote: Total 21354 (delta 16466), reused 19888 (delta 15066)
 Receiving objects: 100% (21354/21354), 176.42 MiB | 1.22 MiB/s, done.
 Resolving deltas: 100% (16466/16466), done.
 Debug: Remote helper: - lock 
 /tmp/ValyriaTear/.git/objects/pack/pack-b6f360ab28b5078a9aefafe1c4144e6c7782c317.keep
 Debug: Remote helper: Waiting...
 Debug: Remote helper: - connectivity-ok
 Debug: Remote helper: Waiting...
 Debug: Remote helper: - 
 Checking connectivity... done.
 Debug: Disconnecting.
 warning: in http_cleanup
 warning: calling curl_multi_remove_handle
 warning: calling curl_easy_cleanup on slot
 warning: curl_easy_cleanup done
 warning: calling curl_easy_cleanup on default
 warning: curl_easy_cleanup done
 warning: calling curl_multi_cleanup
 error: git-remote-https died of signal 13

Thanks. I'm having trouble reproducing the SIGPIPE locally, but I am
able to see via strace the write we make in curl_multi_cleanup. The
call stack is:

  curl_multi_cleanup
- close_all_connections
  - Curl_disconnect
- Curl_ossl_close
  ...

Daniel, does the call to Curl_disconnect need to be wrapped with
sigpipe_ignore/reset, similar to 7d80ed64e435155?

-Peff
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Daniel Stenberg]

On Mon, 25 Nov 2013, Jeff King wrote:

Thanks. I'm having trouble reproducing the SIGPIPE locally, but I am able to 
see via strace the write we make in curl_multi_cleanup. The call stack is:


 curl_multi_cleanup
   - close_all_connections
 - Curl_disconnect
   - Curl_ossl_close
 ...

Daniel, does the call to Curl_disconnect need to be wrapped with 
sigpipe_ignore/reset, similar to 7d80ed64e435155?


Yes. It very much looks like that. The SSL closing is what was the problem I 
had to adress.


But I then decided that if a 3rd library has one way to generate SIGPIPE it 
may very well have another in a separate spot so I decided to do the wrap at 
the top level immediately in the entry point when getting called by the 
application. Following that, the SIGPIPE ignore/restore should rather be made 
in curl_multi_cleanup.


--

 / daniel.haxx.se
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

error: git-remote-https died of signal 13

[Stefan Beller]

Hi,

Using the latest git version (1.8.5.rc3), I get this
this warning/error:

sb@sb:/tmp$ git clone https://github.com/Bertram25/ValyriaTear.git
Cloning into 'ValyriaTear'...
remote: Counting objects: 21346, done.
remote: Compressing objects: 100% (6307/6307), done.
remote: Total 21346 (delta 16463), reused 19820 (delta 15000)
Receiving objects: 100% (21346/21346), 176.39 MiB | 445.00 KiB/s, done.
Resolving deltas: 100% (16463/16463), done.
Checking connectivity... done.
error: git-remote-https died of signal 13

However the repository seems to be cloned fine.
I can clone different repos from github and they are working fine
without the error.
Would that be an issue on my side or on githubs side?

Stefan
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: error: git-remote-https died of signal 13

[Jeff King]

On Sat, Nov 23, 2013 at 05:36:36PM +0100, Stefan Beller wrote:

 sb@sb:/tmp$ git clone https://github.com/Bertram25/ValyriaTear.git
 Cloning into 'ValyriaTear'...
 remote: Counting objects: 21346, done.
 remote: Compressing objects: 100% (6307/6307), done.
 remote: Total 21346 (delta 16463), reused 19820 (delta 15000)
 Receiving objects: 100% (21346/21346), 176.39 MiB | 445.00 KiB/s, done.
 Resolving deltas: 100% (16463/16463), done.
 Checking connectivity... done.
 error: git-remote-https died of signal 13
 
 However the repository seems to be cloned fine.
 I can clone different repos from github and they are working fine
 without the error.
 Would that be an issue on my side or on githubs side?

Almost certainly on your side. 13 is SIGPIPE, so git-remote-https is
trying to write something but the other side of the pipe has hung up.
This might be a race condition in the transport-helper protocol, where
we've had similar problems before.

It doesn't reproduce here for me. Can you reproduce it consistently on
this repo? I would not be at all surprised if it is intermittent.

If you can reproduce, it would be interesting to see the output with
GIT_TRANSPORT_HELPER_DEBUG=1, or even with strace -f. That could at
least tell us what it was trying to write (and to where) when it got the
SIGPIPE.

-Peff
--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html