Hey Willy,
On 2024-04-05 19:44, Willy Tarreau wrote:
Thanks a lot for these details. One thing to have in mind that could
explain that you have not observed this on other servers is that we're
using plain HTTP, we haven't deployed the git-server stuff, so maybe
it triggers a different object transfer sequence and/or code path.
You're definitely onto something here, by forcing git client to use HTTP
dumb (GIT_SMART_HTTP=0), it's also failing against my own endpoint using
HAProxy 2.8.7-1a82cdf, although with different symptoms, clone end up
hanging, although (still with HTTP dumb forced) work just fine with curl
8.6.0 with HTTP2 enabled, or curl 8.7.1 with HTTP2 disabled, so that's
definitely an HTTP2 issue and likely not an HAProxy one. Let me know if
you want me to drop haproxy list form recipients.
Bisecting curl again in this situation points to a different first bad
commit: 0dc036225b30 ("HTTP/2: write response directly")
(https://github.com/curl/curl/commit/0dc036225b3013bf994da81fa44571bcfcecbe92)
Note this issue only appears when curl is compiled with HTTP2 enabled.
I'm
quite sure git.haproxy.org is running on bleeding edge HAProxy, which
might
not be supporting changes brought by 463472a2d6e3, however I'm curious
here
to understand protocol corruption.
If you want to make some tests, we can synchronize.
I've done more than enough for this week, I'm off for today, so I'm
definitely free at the moment :)
Among the easy stuff
we can try on the haproxy side are:
- restart with the stable version in case it has any effect
- disable H2 (not sure if that's doable in git-clone, but we can also
easily open an extra port for you to test)
I haven't found a way to prevent git from using HTTP2, curl is small
enough to recompile quickly to test with different features enabled.
I've been only observing this issue with HTTP2 indeed, I'm not able to
reproduce initial findings with curl built with HTTP2 disabled at build
time.
Just let us know if you're interested. We can also first wait for
Stefan
and/or Daniel's analysis of a possible cause for the commit you
bisected
above before hacking too much stuff, though :-)
Let's see! Latest digging seems to lead to some issue with libcurl in
this case, I might do some test to drop haproxy from the loop to
eliminate that, but it'll take me a few hours before I can setup lab.
Cheers,
Willy
--
Bertrand
