Re: git clone git.haproxy.git with curl-8.7.1 failing writing received data
On 2024-04-05 20:24, Bertrand Jacquin wrote: Just let us know if you're interested. We can also first wait for Stefan and/or Daniel's analysis of a possible cause for the commit you bisected above before hacking too much stuff, though :-) Let's see! Latest digging seems to lead to some issue with libcurl in this case, I might do some test to drop haproxy from the loop to eliminate that, but it'll take me a few hours before I can setup lab. Well, I'm just stupid, I've a setup now bypassing HAProxy directly targeting nginx. I did tests against H2 with and without TLS, I'm not able to reproduce any of both issue (fail writing received data with curl 8.7.1 and git falling back to use dumb http client, and hang with dumb http client forced) with curl 8.7.1 with H2 enabled Cheers, -- Bertrand
Re: git clone git.haproxy.git with curl-8.7.1 failing writing received data
Hey Willy,
On 2024-04-05 19:44, Willy Tarreau wrote:
Thanks a lot for these details. One thing to have in mind that could
explain that you have not observed this on other servers is that we're
using plain HTTP, we haven't deployed the git-server stuff, so maybe
it triggers a different object transfer sequence and/or code path.
You're definitely onto something here, by forcing git client to use HTTP
dumb (GIT_SMART_HTTP=0), it's also failing against my own endpoint using
HAProxy 2.8.7-1a82cdf, although with different symptoms, clone end up
hanging, although (still with HTTP dumb forced) work just fine with curl
8.6.0 with HTTP2 enabled, or curl 8.7.1 with HTTP2 disabled, so that's
definitely an HTTP2 issue and likely not an HAProxy one. Let me know if
you want me to drop haproxy list form recipients.
Bisecting curl again in this situation points to a different first bad
commit: 0dc036225b30 ("HTTP/2: write response directly")
(https://github.com/curl/curl/commit/0dc036225b3013bf994da81fa44571bcfcecbe92)
Note this issue only appears when curl is compiled with HTTP2 enabled.
I'm
quite sure git.haproxy.org is running on bleeding edge HAProxy, which
might
not be supporting changes brought by 463472a2d6e3, however I'm curious
here
to understand protocol corruption.
If you want to make some tests, we can synchronize.
I've done more than enough for this week, I'm off for today, so I'm
definitely free at the moment :)
Among the easy stuff
we can try on the haproxy side are:
- restart with the stable version in case it has any effect
- disable H2 (not sure if that's doable in git-clone, but we can also
easily open an extra port for you to test)
I haven't found a way to prevent git from using HTTP2, curl is small
enough to recompile quickly to test with different features enabled.
I've been only observing this issue with HTTP2 indeed, I'm not able to
reproduce initial findings with curl built with HTTP2 disabled at build
time.
Just let us know if you're interested. We can also first wait for
Stefan
and/or Daniel's analysis of a possible cause for the commit you
bisected
above before hacking too much stuff, though :-)
Let's see! Latest digging seems to lead to some issue with libcurl in
this case, I might do some test to drop haproxy from the loop to
eliminate that, but it'll take me a few hours before I can setup lab.
Cheers,
Willy
--
Bertrand
Re: git clone git.haproxy.git with curl-8.7.1 failing writing received data
Hi Bertrand!
On Fri, Apr 05, 2024 at 07:27:28PM +0100, Bertrand Jacquin wrote:
> Hi,
>
> For the last few days, I've been unable to git clone
> https://git.haproxy.org/git/haproxy.git with curl-8.7.1, where I'm getting
> the following error:
>
> $ GIT_TRACE=1 git fetch https://git.haproxy.org/git/haproxy.git
> 19:12:01.277740 git.c:463 trace: built-in: git fetch
> https://git.haproxy.org/git/haproxy.git
> 19:12:01.278266 run-command.c:657 trace: run_command: GIT_DIR=.git
> git remote-https https://git.haproxy.org/git/haproxy.git
> https://git.haproxy.org/git/haproxy.git
> 19:12:01.279001 git.c:749 trace: exec: git-remote-https
> https://git.haproxy.org/git/haproxy.git
> https://git.haproxy.org/git/haproxy.git
> 19:12:01.279018 run-command.c:657 trace: run_command:
> git-remote-https https://git.haproxy.org/git/haproxy.git
> https://git.haproxy.org/git/haproxy.git
> fatal: unable to access 'https://git.haproxy.org/git/haproxy.git/': Failed
> writing received data to disk/application
(...)
> Bisecting through changes from curl 8.6.0 to 8.7.1 points me to 463472a2d6e3
> ("lib: move client writer into own source")
> (https://github.com/curl/curl/commit/463472a2d6e3301c1468b5323b856cb67a91f579)
> as the first bad commit.
Thanks a lot for these details. One thing to have in mind that could
explain that you have not observed this on other servers is that we're
using plain HTTP, we haven't deployed the git-server stuff, so maybe
it triggers a different object transfer sequence and/or code path.
> Note this issue only appears when curl is compiled with HTTP2 enabled. I'm
> quite sure git.haproxy.org is running on bleeding edge HAProxy, which might
> not be supporting changes brought by 463472a2d6e3, however I'm curious here
> to understand protocol corruption.
If you want to make some tests, we can synchronize. Among the easy stuff
we can try on the haproxy side are:
- restart with the stable version in case it has any effect
- disable H2 (not sure if that's doable in git-clone, but we can also
easily open an extra port for you to test)
Just let us know if you're interested. We can also first wait for Stefan
and/or Daniel's analysis of a possible cause for the commit you bisected
above before hacking too much stuff, though :-)
Cheers,
Willy
