Re: HTTP/3 -- POST requests not working
On 4/15/22 06:40, Shawn Heisey wrote: The 403 is random. While clicking around in my webmail, going to different folders, I occasionally see a red box that has an error message pop up, an error message I can't recall at the moment. That's when the 403 is logged. I noticed there was another change to a quic src file in the git repo when I did a "git pull" ... so I recompiled and reinstalled. Now the webmail seems to work without problems. One other thing to check ... switching my paste website back to http3 and trying to create a paste with tens of thousands of lines. That wasn't working before, had to switch it back to http2. (insert final jeopardy music) ... And that still isn't working. It just hangs, continually indicating the page is still loading. Same as happened before. I will need to get a packet capture for that problem, and compare the upload with http2 to the upload with http3. Thanks, Shawn
Re: HTTP/3 -- POST requests not working
On 4/15/2022 1:20 AM, Amaury Denoyelle wrote: Hum this is strange. Do you have a way to reproduce it easily ? The 403 is random. While clicking around in my webmail, going to different folders, I occasionally see a red box that has an error message pop up, an error message I can't recall at the moment. That's when the 403 is logged. This is the matching Apache log entr for the haproxy log entry I sent earlier: 127.0.0.1 - - [14/Apr/2022:07:11:15 -0600] "POST /mail/?_task=mail&_action=refresh HTTP/2.0" 403 363 "https://webmail.elyograg.org/mail/?_task=mail&_mbox=Sent; "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.75 Safari/537.36" The 403 is not being generated by haproxy. It is coming from Apache. But this does not happen when the connection from the user to haproxy uses http/2, only when it is http/3. When I am more rested, I can grab a packet capture of the traffic between haproxy and apache. This is simple because that connection is h2c. Otherwise, please know that as QUIC is still in an experimental status, we did not have the time to test various config options with it. Yes, I saw that designation. I am hoping that my experiences and assistance can help make it more stable. If you encounter a recurring bug, I advise you to switch to a simple config file and inspect if the issue is still there. What parts of my config would you suggest taking out? Can you give me an example of such a simple config? Although my config is a little long, most of that is due to long ACLs. I have never really thought of that config as complex. :) Hum we already have encounter this issue because we did not send a CONNECTION_CLOSE on connection closing. Now most cases seems to be fixed but maybe there is still cases where the connection dies without a notification to the client. Do you observe this frequently ? Quite frequently. The browser will stop loading a page that is using http3 and I have to completely close the browser to get it working again, which makes testing more difficult. I have not been able to determine what circumstances trigger the problem. The list of domains that I am serving over http3 has shrunk because POST requests don't always work. On the system where haproxy 2.4.15 is listening on TCP ports and 2.6-dev5 is listening on UDP/443, this is the list of domains: acl http3 var(txn.host) -m end unknown.elyograg.org acl http3 var(txn.host) -m end raspi.elyograg.org acl http3 var(txn.host) -m end raspi1.elyograg.org acl http3 var(txn.host) -m end raspi2.elyograg.org acl http3 var(txn.host) -m end shawnheisey.com On the system where the from-git build is the only instance running, it only advertises http3 for http3test.elyograg.org, a simple PHP script. None of the sites where I have currently enabled http3 use POST requests. The webmail is on the same system as the http3test site. It is not widely used by anyone but me, so it is a perfect testing ground of a complex webapp over http3. I can create a mailbox on that system so you can try things yourself. Please let me know how I can be helpful with further testing. Thanks, Shawn
Re: HTTP/3 -- POST requests not working
On Thu, Apr 14, 2022 at 07:29:20AM -0600, Shawn Heisey wrote:
> On 4/14/22 03:27, Amaury Denoyelle wrote:
> > So to summary, this option should be activated if you only have browsers
> > as client and the traffic is big enough to saturate haproxy queues.
> > I hope this will clarify your thoughts,
> Thanks for that detail. For these setups, I really doubt that there will
> ever be that many requests. Unless I get hit by a DDOS attack. :)
> I am having further problems with http/3. Randomly in the webmail it will
> get a 403 error when accessing a URL for the site. Checking the log
> history, looks like it's on POST requests, which were already problematic.
> I guess removing abortonclose doesn't completely fix POST requests.
> Apr 14 07:11:15 - haproxy[2357875] 199.192.164.74:54388
> [14/Apr/2022:07:11:15.227] web~ be-apache-81/bilbo 2/0/76/-1/85 403 1371 - -
> LHVN 1/1/0/0/0 0/0 {webmail.elyograg.org} "POST
> /mail/?_task=mail&_action=refresh HTTP/3.0"
Hum this is strange. Do you have a way to reproduce it easily ?
Otherwise, please know that as QUIC is still in an experimental status,
we did not have the time to test various config options with it. Indeed,
most of the time we use extra simple config files with only timeout
configuration provided. If you encounter a recurring bug, I advise you
to switch to a simple config file and inspect if the issue is still
there. This will allows us to more precisely understand the nature of
the problem.
> And something that has occasionally happened with anything I've enabled
> http/3 on ... firefox will get into a mode where accessing an http/3 enabled
> site will completely hang, and the only way to get that website working
> again is to close the browser, reopen it, and try again. I've also seen
> this in Chrome, so it doesn't seem to be browser-specific. I do most of my
> testing in Firefox, because on the work machine Chrome has a proxy, which
> won't do http/3.
Hum we already have encounter this issue because we did not send a
CONNECTION_CLOSE on connection closing. Now most cases seems to be fixed
but maybe there is still cases where the connection dies without a
notification to the client. Do you observe this frequently ?
> So for now I've only enabled http3 on a select subset of websites. These
> problems make the webmail not work right. I hope any information I gather
> will speed up development on QUIC so it can move out of experimental status.
> Ah, git pull has revealed a lot of changes, some of which are on quic code.
> I will get that built and installed.
As I already said, QUIC in haproxy is still experimental and we expect a
lot of issues to appear :) so please use it with care and always have a
backup solution. We commit frequently changes related to QUIC but the
todo-list is still big and I do not think the latest changes will have
an improvement on your reported issues. But still, it is always useful
to have users ready to test it and report bugs, in fact we already have
fix some thanks to you. Just know that some bugs may take some time
before being addressed on our side.
Thanks,
--
Amaury Denoyelle
Re: HTTP/3 -- POST requests not working
On 4/14/22 03:27, Amaury Denoyelle wrote:
So to summary, this option should be activated if you only have browsers
as client and the traffic is big enough to saturate haproxy queues.
I hope this will clarify your thoughts,
Thanks for that detail. For these setups, I really doubt that there
will ever be that many requests. Unless I get hit by a DDOS attack. :)
I am having further problems with http/3. Randomly in the webmail it
will get a 403 error when accessing a URL for the site. Checking the
log history, looks like it's on POST requests, which were already
problematic. I guess removing abortonclose doesn't completely fix POST
requests.
Apr 14 07:11:15 - haproxy[2357875] 199.192.164.74:54388
[14/Apr/2022:07:11:15.227] web~ be-apache-81/bilbo 2/0/76/-1/85 403 1371
- - LHVN 1/1/0/0/0 0/0 {webmail.elyograg.org} "POST
/mail/?_task=mail&_action=refresh HTTP/3.0"
And something that has occasionally happened with anything I've enabled
http/3 on ... firefox will get into a mode where accessing an http/3
enabled site will completely hang, and the only way to get that website
working again is to close the browser, reopen it, and try again. I've
also seen this in Chrome, so it doesn't seem to be browser-specific. I
do most of my testing in Firefox, because on the work machine Chrome has
a proxy, which won't do http/3.
So for now I've only enabled http3 on a select subset of websites.
These problems make the webmail not work right. I hope any information
I gather will speed up development on QUIC so it can move out of
experimental status.
Ah, git pull has revealed a lot of changes, some of which are on quic
code. I will get that built and installed.
Thanks,
Shawn
Re: HTTP/3 -- POST requests not working
On Wed, Apr 13, 2022 at 08:16:06AM -0600, Shawn Heisey wrote:
> On 4/13/22 02:42, Amaury Denoyelle wrote:
> > Ok this seems related to 'option abortonclose'. Without this, I do not
> > have a 400 error. Can you confirm me this behavior on your side please ?
>
> If I remove that, it works. I can have my webmail served via http/3 and
> login still works, which it didn't before because all POST requests were
> getting a 400.
> What am I losing or gaining by removing abortonclose? I read the docs on
> it, and wasn't able to work out whether I want it or not. Will this failure
> when configuring abortonclose be considered a bug?
Currently, this option is buggy when used with a QUIC listener. This is
indeed a bug and will be fixed in a future patch.
> The reasons for that option being in the config are lost to the antiquities
> of time. Most of my config is stolen from a setup I once built at an old
> ${DAYJOB} that was actually doing load balancing. For my personal setups, I
> mainly use haproxy for SSL offloading and making multiple private-side web
> servers available through a single public IP address.
By default, connection are kept opened in haproxy side even if the
client has closed it. This is required because it's impossible for
haproxy to detect a full close or only a sending channel close. In the
last case, haproxy can still deliver the response and then fully close
the connection. This is conform to the default http specification and is
useful when dealing with clients tools such as netcat which rely heavily
on this behavior.
If all your clients are browsers and the traffic is enough to fill the
haproxy queue, it is useful to activate abortonclose. When a user hits
the "STOP" loading page of the browser, the connection is immediatly
close on haproxy, which liberate some resources to handle other
connections.
So to summary, this option should be activated if you only have browsers
as client and the traffic is big enough to saturate haproxy queues.
I hope this will clarify your thoughts,
--
Amaury Denoyelle
Re: HTTP/3 -- POST requests not working
On 4/13/22 02:42, Amaury Denoyelle wrote:
Ok this seems related to 'option abortonclose'. Without this, I do not
have a 400 error. Can you confirm me this behavior on your side please ?
If I remove that, it works. I can have my webmail served via http/3 and
login still works, which it didn't before because all POST requests were
getting a 400.
What am I losing or gaining by removing abortonclose? I read the docs
on it, and wasn't able to work out whether I want it or not. Will this
failure when configuring abortonclose be considered a bug?
The reasons for that option being in the config are lost to the
antiquities of time. Most of my config is stolen from a setup I once
built at an old ${DAYJOB} that was actually doing load balancing. For
my personal setups, I mainly use haproxy for SSL offloading and making
multiple private-side web servers available through a single public IP
address.
Thanks,
Shawn
Re: HTTP/3 -- POST requests not working
On Wed, Apr 13, 2022 at 10:36:00AM +0200, Amaury Denoyelle wrote: > On Tue, Apr 12, 2022 at 10:30:05AM -0600, Shawn Heisey wrote: > > On 4/12/22 09:45, Amaury Denoyelle wrote: > > > After much analysis of the code, it may be useful to have a run with the > > > stream traces as well : > > > $ trace stream sink buf0; trace stream level developer; trace stream > > > verbosity clean; trace stream start now > > All 3 traces enabled, this time it should only be for one connection, I > > think the last one there were two: > > https://paste.elyograg.org/view/a06556ef > > > You can also provide us your haproxy config file, just remove any > > > sensitive info if you have any. > > https://paste.elyograg.org/view/3f884d8f > > The acl lines with redacted info are not being matched for the connections > > that don't work, so it should have no connection to this problem. > I'm able to reproduce your issue with your config file. I can now > investigate the problem. I will let you know about any update, thanks. Ok this seems related to 'option abortonclose'. Without this, I do not have a 400 error. Can you confirm me this behavior on your side please ? -- Amaury Denoyelle
Re: HTTP/3 -- POST requests not working
On Tue, Apr 12, 2022 at 10:30:05AM -0600, Shawn Heisey wrote: > On 4/12/22 09:45, Amaury Denoyelle wrote: > > After much analysis of the code, it may be useful to have a run with the > > stream traces as well : > > $ trace stream sink buf0; trace stream level developer; trace stream > > verbosity clean; trace stream start now > All 3 traces enabled, this time it should only be for one connection, I > think the last one there were two: > https://paste.elyograg.org/view/a06556ef > > You can also provide us your haproxy config file, just remove any > > sensitive info if you have any. > https://paste.elyograg.org/view/3f884d8f > The acl lines with redacted info are not being matched for the connections > that don't work, so it should have no connection to this problem. I'm able to reproduce your issue with your config file. I can now investigate the problem. I will let you know about any update, thanks. -- Amaury Denoyelle
Re: [EXTERNAL] Re: HTTP/3 -- POST requests not working
On 4/12/22 19:20, Shawn Heisey wrote: https://paste.elyograg.org/view/bd5df44d I repeated it, this time issuing the stream traces first, just in case the order of the trace commands might matter. THat's the sort of thing that shouldn't matter, https://paste.elyograg.org/view/22ddec0a And then I did it one more time, this time ONLY doing the stream trace: https://paste.elyograg.org/view/75b494d5 And here's haproxy -vv: https://paste.elyograg.org/view/46c7c58a Thanks, Shawn
Re: [EXTERNAL] Re: HTTP/3 -- POST requests not working
On 4/12/22 16:06, Frederic Lecaille wrote: so this command: $ echo "trace quic sink buf0; trace quic level developer; trace quic verbosity clean; trace quic start now; trace qmux sink buf0; trace qmux level developer; trace qmux verbosity minimal; trace qmux start now; trace stream sink buf0; trace stream level developer; trace stream verbosity clean; trace stream start now" | socat stdio should work (tested after haven copied and pasted from this mail). What I was doing before doing is an interactive session started with this command that I found somewhere on the Internet: expect -c 'log_user 0;spawn sudo rlwrap -l haproxy_stats_socket.log -S "haproxy@bilbo> " -c socat /etc/haproxy/stats.socket stdio; send "prompt\n"; interact' The interactive session times out very quickly, so I have to have everything ready to go. Before I do anything, I restart haproxy so I am beginning from a clean state. Piping commands through socat seems like a better option, so I divided that one echo command up into 12 separate commands, and you can see the result here: https://paste.elyograg.org/view/bd5df44d It looks to me like there are still no stream traces, and as you can see from the commands I did, I did enable it. I wonder if that's the whole problem ... it isn't sending anything to stream? I have very little knowledge about haproxy internals, so I am just guessing. So you don't have to go looking for it, this is what I am doing to reproduce the problem. This should work from anywhere on the Internet, you can do the exact same command... you just need docker and an Internet connnection: sudo docker run -it --rm ymuski/curl-http3 curl -v https://http3test.elyograg.org -X POST -d " " --http3 Change "--http3" to "--http2" and the request will succeed. Thanks, Shawn
Re: [EXTERNAL] Re: HTTP/3 -- POST requests not working
On 4/12/22 22:42, Shawn Heisey wrote:
>> Please, you could you double check on your side the "stream" traces are
>> correctly enabled? Also ensure you provide use with traces dumped by
>> haproxy when you validate the PHP form.
>
>
> These are the trace commands that I am sending to the stats socket:
>
> trace quic sink buf0; trace quic level developer; trace quic verbosity
> clean; trace quic start now
> trace qmux sink buf0; trace qmux level developer; trace qmux verbosity
> minimal; trace qmux start now
> trace stream sink buf0; trace stream level developer; trace stream
> verbosity clean; trace stream start now
I do not exactly how you send your commands to the CLI, but for instance
if I copy and paste your three commands above from an interactive prompt
(see "prompt" CLI command) I have to press two times enter to ensure all
three commands have been taken into an account.
Here is an extract of the documentation about how to send command to the
haproxy CLI:
9.3. Unix Socket commands
The non-interactive mode is the default when socat connects to the
socket. In this mode, a single line may be sent. It is processed as a
whole, responses are sent back, and the connection closes after the end
of the response. This is the mode that scripts and monitoring tools use.
It is possible to send multiple commands in this mode, they need to be
delimited by a semi-colon (';'). For
example :
# echo "show info;show stat;show table" | socat /var/run/haproxy stdio
so this command:
$ echo "trace quic sink buf0; trace quic level developer; trace quic
verbosity clean; trace quic start now; trace qmux sink buf0; trace qmux
level developer; trace qmux verbosity minimal; trace qmux start now;
trace stream sink buf0; trace stream level developer; trace stream
verbosity clean; trace stream start now" | socat stdio
should work (tested after haven copied and pasted from this mail).
Re: [EXTERNAL] Re: HTTP/3 -- POST requests not working
On 4/12/22 14:07, Frederic Lecaille wrote: Please, you could you double check on your side the "stream" traces are correctly enabled? Also ensure you provide use with traces dumped by haproxy when you validate the PHP form. These are the trace commands that I am sending to the stats socket: trace quic sink buf0; trace quic level developer; trace quic verbosity clean; trace quic start now trace qmux sink buf0; trace qmux level developer; trace qmux verbosity minimal; trace qmux start now trace stream sink buf0; trace stream level developer; trace stream verbosity clean; trace stream start now I am sure that I have pressed enter after the last line. If I have copied that third line correctly, then nothing is being logged for stream. The "index.php" file is not equipped to deal with a POST request, but apache doesn't log anything, so I don't think it's ever getting to PHP. It correctly executes the script when using http2. The script just checks a header that I am setting in haproxy, to say "YES" or "NO" depending on whether the browser was doing http3. https://http3test.elyograg.org/ If you load the page in Firefox, you need to do a regular page reload to get it doing http3. In Chrome, it needs to be a shift-reload. If you do vice-versa, it will never switch to http3. I wonder which behavior could be considered a bug. I expect that there is probably no set release data for 2.6.0, but is there a ballpark date when it MIGHT be ready for release? Thanks, Shawn
Re: [EXTERNAL] Re: HTTP/3 -- POST requests not working
Hello Shawn, On 4/12/22 18:30, Shawn Heisey wrote: > On 4/12/22 09:45, Amaury Denoyelle wrote: >> After much analysis of the code, it may be useful to have a run with the >> stream traces as well : >> $ trace stream sink buf0; trace stream level developer; trace stream >> verbosity clean; trace stream start now > > All 3 traces enabled, this time it should only be for one connection, I > think the last one there were two: It's ok for the quic and qmux parts. But I do not seen any trace for the stream part. You should see such a traces when your h3 client open a stream (truncated here, without timestamp). Note the second field of the traces: "quic", "qmux", "stream" [04|quic|2|t_quic.c:2543] RX frame : qc@0x7f56cc3c08e0 STREAM_A uni=1 fin=0 id=10 off=0 len=1 [04|qmux|5|ux_quic.c:101] qcs_new(): entering : qcc=0x7f56cc376e60(F) [04|qmux|5|ux_quic.c:144] qcs_new(): leaving : qcc=0x7f56cc376e60(F) qcs=0x7f56cc35cf20(6) [04|qmux|5|ux_quic.c:101] qcs_new(): entering : qcc=0x7f56cc376e60(F) [04|qmux|5|ux_quic.c:144] qcs_new(): leaving : qcc=0x7f56cc376e60(F) qcs=0x7f56cc35af20(10) . . . [04|stream|5|/stream.c:348] stream_new(): entering [04|stream|5|/stream.c:566] stream_new(): leaving : [2,HTX] SI=(EST,INI) . . . Please, you could you double check on your side the "stream" traces are correctly enabled? Also ensure you provide use with traces dumped by haproxy when you validate the PHP form. Thank you.
Re: HTTP/3 -- POST requests not working
On 4/12/22 09:45, Amaury Denoyelle wrote: After much analysis of the code, it may be useful to have a run with the stream traces as well : $ trace stream sink buf0; trace stream level developer; trace stream verbosity clean; trace stream start now All 3 traces enabled, this time it should only be for one connection, I think the last one there were two: https://paste.elyograg.org/view/a06556ef You can also provide us your haproxy config file, just remove any sensitive info if you have any. https://paste.elyograg.org/view/3f884d8f The acl lines with redacted info are not being matched for the connections that don't work, so it should have no connection to this problem. Thanks, Shawn
Re: HTTP/3 -- POST requests not working
On Tue, Apr 12, 2022 at 05:26:41PM +0200, Amaury Denoyelle wrote: > On Tue, Apr 12, 2022 at 08:01:59AM -0600, Shawn Heisey wrote: > > On 4/12/22 02:22, Amaury Denoyelle wrote: > > > then you can display the traces with the following command : > > > $ show events buf0 > > > > For the h3 layer, the trace mechanism is not currently implemented. You > > > should instead recompile your haproxy binary with the DEBUG options : > > > -DDEBUG_H3 -DDEBUG_QPACK > > > and watch for the stderr output for your process. > > I got some traces. URL below for accessing it. There's a LOT of data. > > https://paste.elyograg.org/view/8678c875 > > I didn't do the H3 debug yet, if you still need it after looking at the > > traces, let me know. You'll also need to tell me how to make those debugs > > active when I build haproxy. > We analyzed the output of the traces but there is nothing of particular > interest here. What may be useful is to activate the H3 traces. > However, I notice that the -dev5 does not compile with H3 traces > activated. So first, you have to apply the attached patch before > recompiling. To do this, save the attachment and execute the following > command in your source folder : > $ patch -p1 < 0001-BUG-MINOR-h3-fix-build-with-DEBUG_H3.patch > Alternatively, you can also use the latest master with contains the > mentionned patch. However, I think it's safer to stay on the -dev5 tag > for the moment. > Once the patch is applied, recompile your haproxy binary with DEBUG_H3 > and DEBUG_QPACK flags. Here is a make example, complete it with your > usual set of flags : > $ make TARGET=linux-glibc USE_QUIC=1 DEBUG="-DDEBUG_H3 -DDEBUG_QPACK" ... > You can now restart your process and watch the stderr output and report > it to us. > Again, thank you very much for your help, Hi again, After much analysis of the code, it may be useful to have a run with the stream traces as well : $ trace stream sink buf0; trace stream level developer; trace stream verbosity clean; trace stream start now You can also provide us your haproxy config file, just remove any sensitive info if you have any. Thanks, -- Amaury Denoyelle
Re: HTTP/3 -- POST requests not working
On 4/12/22 08:01, Shawn Heisey wrote: I didn't do the H3 debug yet, if you still need it after looking at > the traces, let me know. You'll also need to tell me how to make > those debugs active when I build haproxy. I figured this out. I put those debug definitions in a DEBUG="stuff" parameter on the build. The 2.6-dev5 build fails. And it looks like this is known ... a change was pushed a short time ago that says it fixes building with DEBUG_H3. Building from the github repo worked and now my version is 2.6-dev5-bb9704-38. I redirected stdout and stderr to files with my systemd service definition. Here's what got sent to stderr: https://paste.elyograg.org/view/ebe7a659 Thanks, Shawn
Re: HTTP/3 -- POST requests not working
On Tue, Apr 12, 2022 at 08:01:59AM -0600, Shawn Heisey wrote: > On 4/12/22 02:22, Amaury Denoyelle wrote: > > then you can display the traces with the following command : > > $ show events buf0 > > > For the h3 layer, the trace mechanism is not currently implemented. You > > should instead recompile your haproxy binary with the DEBUG options : > > -DDEBUG_H3 -DDEBUG_QPACK > > and watch for the stderr output for your process. > I got some traces. URL below for accessing it. There's a LOT of data. > https://paste.elyograg.org/view/8678c875 > I didn't do the H3 debug yet, if you still need it after looking at the > traces, let me know. You'll also need to tell me how to make those debugs > active when I build haproxy. We analyzed the output of the traces but there is nothing of particular interest here. What may be useful is to activate the H3 traces. However, I notice that the -dev5 does not compile with H3 traces activated. So first, you have to apply the attached patch before recompiling. To do this, save the attachment and execute the following command in your source folder : $ patch -p1 < 0001-BUG-MINOR-h3-fix-build-with-DEBUG_H3.patch Alternatively, you can also use the latest master with contains the mentionned patch. However, I think it's safer to stay on the -dev5 tag for the moment. Once the patch is applied, recompile your haproxy binary with DEBUG_H3 and DEBUG_QPACK flags. Here is a make example, complete it with your usual set of flags : $ make TARGET=linux-glibc USE_QUIC=1 DEBUG="-DDEBUG_H3 -DDEBUG_QPACK" ... You can now restart your process and watch the stderr output and report it to us. Again, thank you very much for your help, -- Amaury Denoyelle >From bb970422546c0db08b244d5e5a12a795a605dd64 Mon Sep 17 00:00:00 2001 From: Amaury Denoyelle Date: Tue, 12 Apr 2022 16:40:52 +0200 Subject: [PATCH] BUG/MINOR: h3: fix build with DEBUG_H3 qcs by_id field has been replaced by a new field named "id". Adjust the h3_debug_printf traces. This is the case since the introduction of the qc_stream_desc type. --- src/h3.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/h3.c b/src/h3.c index e39c54135..2d450d8b8 100644 --- a/src/h3.c +++ b/src/h3.c @@ -231,7 +231,7 @@ static int h3_decode_qcs(struct qcs *qcs, int fin, void *ctx) struct buffer *rxbuf = >rx.buf; int ret; - h3_debug_printf(stderr, "%s: STREAM ID: %llu\n", __func__, qcs->by_id.key); + h3_debug_printf(stderr, "%s: STREAM ID: %lu\n", __func__, qcs->id); if (!b_data(rxbuf)) return 0; @@ -333,7 +333,7 @@ static int h3_control_recv(struct h3_uqs *h3_uqs, void *ctx) struct buffer *rxbuf = _uqs->qcs->rx.buf; struct h3 *h3 = ctx; - h3_debug_printf(stderr, "%s STREAM ID: %llu\n", __func__, h3_uqs->qcs->by_id.key); + h3_debug_printf(stderr, "%s STREAM ID: %lu\n", __func__, h3_uqs->qcs->id); if (!b_data(rxbuf)) return 1; -- 2.35.1
Re: HTTP/3 -- POST requests not working
On 4/12/22 02:22, Amaury Denoyelle wrote: then you can display the traces with the following command : $ show events buf0 For the h3 layer, the trace mechanism is not currently implemented. You should instead recompile your haproxy binary with the DEBUG options : -DDEBUG_H3 -DDEBUG_QPACK and watch for the stderr output for your process. I got some traces. URL below for accessing it. There's a LOT of data. https://paste.elyograg.org/view/8678c875 I didn't do the H3 debug yet, if you still need it after looking at the traces, let me know. You'll also need to tell me how to make those debugs active when I build haproxy. Thanks, Shawn
Re: HTTP/3 -- POST requests not working
On Tue, Apr 12, 2022 at 08:14:25AM +0200, Willy Tarreau wrote: > Hi Shawn, > [...] > On Mon, Apr 11, 2022 at 01:05:21PM -0600, Shawn Heisey wrote: > > An astute observer will notice that the backend says 81 while the packet > > capture shows port 82. This is because I changed the port number on the > > server to point to nginx instead of apache, but did not rename the backend. > > My use of nginx is temporary, only using it to eliminate Apache as the > > problem. I know how to configure Apache; it would take me forever to fully > > switch to nginx. > I think that the QUIC team will soon ask you to enable traces on the QUIC > layer, but I'll let them follow up ;-) Hi Shawn, First, thank you for your interest for QUIC in haproxy :) I have quickly tested POST on my side but did not see any issue. To help you diagnostic your problem, you can try to activate the traces on the stats socket. for the transport traces : $ trace quic sink buf0; trace quic level developer; trace quic verbosity clean; trace quic start now and for the mux : $ trace qmux sink buf0; trace qmux level developer; trace qmux verbosity minimal; trace qmux start now then you can display the traces with the following command : $ show events buf0 For the h3 layer, the trace mechanism is not currently implemented. You should instead recompile your haproxy binary with the DEBUG options : -DDEBUG_H3 -DDEBUG_QPACK and watch for the stderr output for your process. Thanks again for your help, Regards, -- Amaury Denoyelle
Re: HTTP/3 -- POST requests not working
Shawn, I wonder if there a test suite similar to h2check (which is http2 and hpack). I saw some quic conformance test reported on release notes, but it seem did not catch POST issue. Should we try to report to them? On Tue, Apr 12, 2022, 4:45 AM Shawn Heisey wrote: > On 4/11/2022 4:51 PM, Shawn Heisey wrote: > > I have a more reliable way of reproducing the problem. Finally found > > a way to get a version of curl that supports http3. On a machine with > > docker and Internet connectivity, run this command: > > > > sudo docker run -it --rm ymuski/curl-http3 curl -v > > https://raspi2.elyograg.org -X POST -d " " --http3 > > The URL above is on the system where TCP is handled by haproxy 2.4.15 > and UDP by 2.6-dev5, a fact that might taint the experiment. The > following test is to a single haproxy 2.6-dev5 instance handling all > traffic: > > sudo docker run -it --rm ymuski/curl-http3 curl -v > https://http3test.elyograg.org -X POST -d " " --http3 > > Again, if http3 is replaced with http2, the request works. > > Thanks, > Shawn > > >
Re: HTTP/3 -- POST requests not working
Hi Shawn,
On Mon, Apr 11, 2022 at 01:05:21PM -0600, Shawn Heisey wrote:
> On 4/11/22 12:16, Shawn Heisey wrote:
> > Two different browsers have the same problem, so I am currently
> > speculating that it's an issue with haproxy or apache. I think I can
> > install nginx without too much trouble ... I have almost no experience
> > with it, so I get to learn something new.
>
>
> Followup:
>
> I changed the backend to talk to nginx instead of apache. It still throws a
> 400 when the POST is done via http/3. It was an adventure trying to figure
> out how to allow POST requests in nginx. The 400 appears to be coming from
> haproxy. There is nothing in the logs on apache or nginx for those errors.
>
> Here is a screenshot of the backend traffic from that POST seen in
> wireshark. 192.168.1.200 is the machine with haproxy, 192.168.1.165 is the
> raspberry pi running nginx.
>
> https://www.dropbox.com/s/h2d8b6xmgpvphfb/Screenshot%20from%202022-04-11%2012-49-05.png?dl=0
>
> What I see there is that haproxy sends the POST, nginx sends an ACK, and
> then haproxy resets the connection. Am I interpreting that correctly?
Yes, but this alone could be caused by different reasons, it simply
indicates an abort from haproxy.
> This is the matching entry from the haproxy log:
>
> Apr 11 12:47:08 - haproxy[1486593] 192.168.1.219:36723
> [11/Apr/2022:12:47:05.189] web be-raspi2-81/legolas 3053/0/1/-1/3053 400 206
> - - CHVN 1/1/0/0/0 0/0 {raspi2.elyograg.org} "POST / HTTP/3.0"
"CH" here indicates a client error before the server responds. This most
likely indicates that some contents were not correctly decoded from the
request body, or that the end of request was mistaken for a client abort.
> I noticed the frontend (web) is missing the ~ character in that log entry.
> That is probably related to the fact that ssl_fc is not set for HTTP/3
> connections.
Not for this one :-) For the logs there was an exact check for the
regular SSL transport layer. It's now fixed to check if the transport
layer supports SSL. This one, as well as the ssl_* sample fetch functions
were now fixed in latest git if you're interested.
> An astute observer will notice that the backend says 81 while the packet
> capture shows port 82. This is because I changed the port number on the
> server to point to nginx instead of apache, but did not rename the backend.
> My use of nginx is temporary, only using it to eliminate Apache as the
> problem. I know how to configure Apache; it would take me forever to fully
> switch to nginx.
I think that the QUIC team will soon ask you to enable traces on the QUIC
layer, but I'll let them follow up ;-)
Cheers,
Willy
Re: HTTP/3 -- POST requests not working
On 4/11/2022 4:51 PM, Shawn Heisey wrote: I have a more reliable way of reproducing the problem. Finally found a way to get a version of curl that supports http3. On a machine with docker and Internet connectivity, run this command: sudo docker run -it --rm ymuski/curl-http3 curl -v https://raspi2.elyograg.org -X POST -d " " --http3 The URL above is on the system where TCP is handled by haproxy 2.4.15 and UDP by 2.6-dev5, a fact that might taint the experiment. The following test is to a single haproxy 2.6-dev5 instance handling all traffic: sudo docker run -it --rm ymuski/curl-http3 curl -v https://http3test.elyograg.org -X POST -d " " --http3 Again, if http3 is replaced with http2, the request works. Thanks, Shawn
Re: HTTP/3 -- POST requests not working
On 4/11/2022 1:05 PM, Shawn Heisey wrote: I changed the backend to talk to nginx instead of apache. It still throws a 400 when the POST is done via http/3. It was an adventure trying to figure out how to allow POST requests in nginx. The 400 appears to be coming from haproxy. There is nothing in the logs on apache or nginx for those errors. Second followup: I have a more reliable way of reproducing the problem. Finally found a way to get a version of curl that supports http3. On a machine with docker and Internet connectivity, run this command: sudo docker run -it --rm ymuski/curl-http3 curl -v https://raspi2.elyograg.org -X POST -d " " --http3 That curl sends an ALPN of h3-29,h3-28,h3-27 so one of those will need to be enabled on the alpn keyword in the haproxy config. I have done this for my config at the URL above. It will reliably show a 400 error. If the "--http3" is changed to "--http2" the request will work. Thanks, Shawn
Re: HTTP/3 -- POST requests not working
On 4/11/22 12:16, Shawn Heisey wrote:
Two different browsers have the same problem, so I am currently
speculating that it's an issue with haproxy or apache. I think I can
install nginx without too much trouble ... I have almost no experience
with it, so I get to learn something new.
Followup:
I changed the backend to talk to nginx instead of apache. It still
throws a 400 when the POST is done via http/3. It was an adventure
trying to figure out how to allow POST requests in nginx. The 400
appears to be coming from haproxy. There is nothing in the logs on
apache or nginx for those errors.
Here is a screenshot of the backend traffic from that POST seen in
wireshark. 192.168.1.200 is the machine with haproxy, 192.168.1.165 is
the raspberry pi running nginx.
https://www.dropbox.com/s/h2d8b6xmgpvphfb/Screenshot%20from%202022-04-11%2012-49-05.png?dl=0
What I see there is that haproxy sends the POST, nginx sends an ACK, and
then haproxy resets the connection. Am I interpreting that correctly?
This is the matching entry from the haproxy log:
Apr 11 12:47:08 - haproxy[1486593] 192.168.1.219:36723
[11/Apr/2022:12:47:05.189] web be-raspi2-81/legolas 3053/0/1/-1/3053 400
206 - - CHVN 1/1/0/0/0 0/0 {raspi2.elyograg.org} "POST / HTTP/3.0"
I noticed the frontend (web) is missing the ~ character in that log
entry. That is probably related to the fact that ssl_fc is not set for
HTTP/3 connections.
An astute observer will notice that the backend says 81 while the packet
capture shows port 82. This is because I changed the port number on the
server to point to nginx instead of apache, but did not rename the
backend. My use of nginx is temporary, only using it to eliminate
Apache as the problem. I know how to configure Apache; it would take me
forever to fully switch to nginx.
Thanks,
Shawn
