Re: git clone git.haproxy.git with curl-8.7.1 failing writing received data

2024-04-05 Thread Bertrand Jacquin

On 2024-04-05 20:24, Bertrand Jacquin wrote:

Just let us know if you're interested. We can also first wait for 
Stefan
and/or Daniel's analysis of a possible cause for the commit you 
bisected

above before hacking too much stuff, though :-)


Let's see! Latest digging seems to lead to some issue with libcurl in 
this case, I might do some test to drop haproxy from the loop to 
eliminate that, but it'll take me a few hours before I can setup lab.


Well, I'm just stupid, I've a setup now bypassing HAProxy directly 
targeting nginx. I did tests against H2 with and without TLS, I'm not 
able to reproduce any of both issue (fail writing received data with 
curl 8.7.1 and git falling back to use dumb http client, and hang with 
dumb http client forced) with curl 8.7.1 with H2 enabled


Cheers,

--
Bertrand



Re: git clone git.haproxy.git with curl-8.7.1 failing writing received data

2024-04-05 Thread Bertrand Jacquin

Hey Willy,

On 2024-04-05 19:44, Willy Tarreau wrote:


Thanks a lot for these details. One thing to have in mind that could
explain that you have not observed this on other servers is that we're
using plain HTTP, we haven't deployed the git-server stuff, so maybe
it triggers a different object transfer sequence and/or code path.


You're definitely onto something here, by forcing git client to use HTTP 
dumb (GIT_SMART_HTTP=0), it's also failing against my own endpoint using 
HAProxy 2.8.7-1a82cdf, although with different symptoms, clone end up 
hanging, although (still with HTTP dumb forced) work just fine with curl 
8.6.0 with HTTP2 enabled, or curl 8.7.1 with HTTP2 disabled, so that's 
definitely an HTTP2 issue and likely not an HAProxy one. Let me know if 
you want me to drop haproxy list form recipients.


Bisecting curl again in this situation points to a different first bad 
commit: 0dc036225b30 ("HTTP/2: write response directly") 
(https://github.com/curl/curl/commit/0dc036225b3013bf994da81fa44571bcfcecbe92)


Note this issue only appears when curl is compiled with HTTP2 enabled. 
I'm
quite sure git.haproxy.org is running on bleeding edge HAProxy, which 
might
not be supporting changes brought by 463472a2d6e3, however I'm curious 
here

to understand protocol corruption.


If you want to make some tests, we can synchronize.


I've done more than enough for this week, I'm off for today, so I'm 
definitely free at the moment :)



Among the easy stuff
we can try on the haproxy side are:
  - restart with the stable version in case it has any effect
  - disable H2 (not sure if that's doable in git-clone, but we can also
easily open an extra port for you to test)


I haven't found a way to prevent git from using HTTP2, curl is small 
enough to recompile quickly to test with different features enabled. 
I've been only observing this issue with HTTP2 indeed, I'm not able to 
reproduce initial findings with curl built with HTTP2 disabled at build 
time.


Just let us know if you're interested. We can also first wait for 
Stefan
and/or Daniel's analysis of a possible cause for the commit you 
bisected

above before hacking too much stuff, though :-)


Let's see! Latest digging seems to lead to some issue with libcurl in 
this case, I might do some test to drop haproxy from the loop to 
eliminate that, but it'll take me a few hours before I can setup lab.



Cheers,
Willy


--
Bertrand



Re: git clone git.haproxy.git with curl-8.7.1 failing writing received data

2024-04-05 Thread Willy Tarreau
Hi Bertrand!

On Fri, Apr 05, 2024 at 07:27:28PM +0100, Bertrand Jacquin wrote:
> Hi,
> 
> For the last few days, I've been unable to git clone
> https://git.haproxy.org/git/haproxy.git with curl-8.7.1, where I'm getting
> the following error:
> 
>   $ GIT_TRACE=1 git fetch https://git.haproxy.org/git/haproxy.git
>   19:12:01.277740 git.c:463   trace: built-in: git fetch
> https://git.haproxy.org/git/haproxy.git
>   19:12:01.278266 run-command.c:657   trace: run_command: GIT_DIR=.git
> git remote-https https://git.haproxy.org/git/haproxy.git
> https://git.haproxy.org/git/haproxy.git
>   19:12:01.279001 git.c:749   trace: exec: git-remote-https
> https://git.haproxy.org/git/haproxy.git
> https://git.haproxy.org/git/haproxy.git
>   19:12:01.279018 run-command.c:657   trace: run_command:
> git-remote-https https://git.haproxy.org/git/haproxy.git
> https://git.haproxy.org/git/haproxy.git
>   fatal: unable to access 'https://git.haproxy.org/git/haproxy.git/': Failed
> writing received data to disk/application
(...)
> Bisecting through changes from curl 8.6.0 to 8.7.1 points me to 463472a2d6e3
> ("lib: move client writer into own source") 
> (https://github.com/curl/curl/commit/463472a2d6e3301c1468b5323b856cb67a91f579)
> as the first bad commit.

Thanks a lot for these details. One thing to have in mind that could
explain that you have not observed this on other servers is that we're
using plain HTTP, we haven't deployed the git-server stuff, so maybe
it triggers a different object transfer sequence and/or code path.

> Note this issue only appears when curl is compiled with HTTP2 enabled. I'm
> quite sure git.haproxy.org is running on bleeding edge HAProxy, which might
> not be supporting changes brought by 463472a2d6e3, however I'm curious here
> to understand protocol corruption.

If you want to make some tests, we can synchronize. Among the easy stuff
we can try on the haproxy side are:
  - restart with the stable version in case it has any effect
  - disable H2 (not sure if that's doable in git-clone, but we can also
easily open an extra port for you to test)

Just let us know if you're interested. We can also first wait for Stefan
and/or Daniel's analysis of a possible cause for the commit you bisected
above before hacking too much stuff, though :-)

Cheers,
Willy



git clone git.haproxy.git with curl-8.7.1 failing writing received data

2024-04-05 Thread Bertrand Jacquin

Hi,

For the last few days, I've been unable to git clone 
https://git.haproxy.org/git/haproxy.git with curl-8.7.1, where I'm 
getting the following error:


  $ GIT_TRACE=1 git fetch https://git.haproxy.org/git/haproxy.git
  19:12:01.277740 git.c:463   trace: built-in: git fetch 
https://git.haproxy.org/git/haproxy.git
  19:12:01.278266 run-command.c:657   trace: run_command: 
GIT_DIR=.git git remote-https https://git.haproxy.org/git/haproxy.git 
https://git.haproxy.org/git/haproxy.git
  19:12:01.279001 git.c:749   trace: exec: git-remote-https 
https://git.haproxy.org/git/haproxy.git 
https://git.haproxy.org/git/haproxy.git
  19:12:01.279018 run-command.c:657   trace: run_command: 
git-remote-https https://git.haproxy.org/git/haproxy.git 
https://git.haproxy.org/git/haproxy.git
  fatal: unable to access 'https://git.haproxy.org/git/haproxy.git/': 
Failed writing received data to disk/application


I'm able to identify anything going off on the client, even though the 
error seems to indicate git could not write data on disk. After doing 
some digging, fetching https://git.haproxy.org/git/haproxy.git using 
curl 8.6.0 works like a charm, however I'm only seeing issue with git 
fetch/clone against git.haproxy.org and no other repo, thus both mailing 
list are made destination of this email.


Here is attached the output of git fetch with GIT_TRACE_CURL=1 in 
git-fetch-haproxy-curl-8.7.1.log


Version of git used on client side:
  git version 2.43.2
  cpu: x86_64
  no commit associated with this build
  sizeof-long: 8
  sizeof-size_t: 8
  shell-path: /bin/sh

Details of functional curl 8.6.0:
  8.6.0 (x86_64-pc-linux-gnu) libcurl/8.6.0 OpenSSL/3.0.13 zlib/1.3 
zstd/1.5.5 c-ares/1.26.0 libidn2/2.3.7 libpsl/0.21.5 nghttp2/1.57.0

  Release-Date: 2024-01-31
  Protocols: dict file http https imap imaps ipfs ipns mqtt pop3 pop3s 
rtsp smtp smtps tftp
  Features: alt-svc AsynchDNS HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile 
libz NTLM PSL SSL threadsafe TLS-SRP UnixSockets zstd


Details of non functional curl 8.7.1
  curl 8.7.1 (x86_64-pc-linux-gnu) libcurl/8.7.1 OpenSSL/3.0.13 zlib/1.3 
brotli/1.1.0 zstd/1.5.5 c-ares/1.26.0 libidn2/2.3.7 libpsl/0.21.5 
libssh2/1.11.0 nghttp2/1.61.0 librtmp/2.3

  Release-Date: 2024-03-27
  Protocols: dict file ftp ftps gopher gophers http https imap imaps 
ipfs ipns mqtt pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp ws 
wss
  Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 
Largefile libz NTLM PSL SSL threadsafe TLS-SRP UnixSockets zstd


Bisecting through changes from curl 8.6.0 to 8.7.1 points me to 
463472a2d6e3 ("lib: move client writer into own source") 
(https://github.com/curl/curl/commit/463472a2d6e3301c1468b5323b856cb67a91f579) 
as the first bad commit.


Note this issue only appears when curl is compiled with HTTP2 enabled. 
I'm quite sure git.haproxy.org is running on bleeding edge HAProxy, 
which might not be supporting changes brought by 463472a2d6e3, however 
I'm curious here to understand protocol corruption.


Cheers,

--
Bertrand

git-fetch-haproxy-curl-8.7.1.log
Description: Binary data