Re: [EXTERNAL] Re: HTTP/3 -- POST requests not working
On 4/12/22 19:20, Shawn Heisey wrote: https://paste.elyograg.org/view/bd5df44d I repeated it, this time issuing the stream traces first, just in case the order of the trace commands might matter. THat's the sort of thing that shouldn't matter, https://paste.elyograg.org/view/22ddec0a And then I did it one more time, this time ONLY doing the stream trace: https://paste.elyograg.org/view/75b494d5 And here's haproxy -vv: https://paste.elyograg.org/view/46c7c58a Thanks, Shawn
Re: [EXTERNAL] Re: HTTP/3 -- POST requests not working
On 4/12/22 16:06, Frederic Lecaille wrote: so this command: $ echo "trace quic sink buf0; trace quic level developer; trace quic verbosity clean; trace quic start now; trace qmux sink buf0; trace qmux level developer; trace qmux verbosity minimal; trace qmux start now; trace stream sink buf0; trace stream level developer; trace stream verbosity clean; trace stream start now" | socat stdio should work (tested after haven copied and pasted from this mail). What I was doing before doing is an interactive session started with this command that I found somewhere on the Internet: expect -c 'log_user 0;spawn sudo rlwrap -l haproxy_stats_socket.log -S "haproxy@bilbo> " -c socat /etc/haproxy/stats.socket stdio; send "prompt\n"; interact' The interactive session times out very quickly, so I have to have everything ready to go. Before I do anything, I restart haproxy so I am beginning from a clean state. Piping commands through socat seems like a better option, so I divided that one echo command up into 12 separate commands, and you can see the result here: https://paste.elyograg.org/view/bd5df44d It looks to me like there are still no stream traces, and as you can see from the commands I did, I did enable it. I wonder if that's the whole problem ... it isn't sending anything to stream? I have very little knowledge about haproxy internals, so I am just guessing. So you don't have to go looking for it, this is what I am doing to reproduce the problem. This should work from anywhere on the Internet, you can do the exact same command... you just need docker and an Internet connnection: sudo docker run -it --rm ymuski/curl-http3 curl -v https://http3test.elyograg.org -X POST -d " " --http3 Change "--http3" to "--http2" and the request will succeed. Thanks, Shawn
write to log file
Hi My configuration do not do what I thought it should do, so I would like to debug it. Old school with "print" commands of acl's, but I cannot find a method to do that. Is it possible and if so what is the command. Regards Henning
Re: [EXTERNAL] Re: HTTP/3 -- POST requests not working
On 4/12/22 22:42, Shawn Heisey wrote:
>> Please, you could you double check on your side the "stream" traces are
>> correctly enabled? Also ensure you provide use with traces dumped by
>> haproxy when you validate the PHP form.
>
>
> These are the trace commands that I am sending to the stats socket:
>
> trace quic sink buf0; trace quic level developer; trace quic verbosity
> clean; trace quic start now
> trace qmux sink buf0; trace qmux level developer; trace qmux verbosity
> minimal; trace qmux start now
> trace stream sink buf0; trace stream level developer; trace stream
> verbosity clean; trace stream start now
I do not exactly how you send your commands to the CLI, but for instance
if I copy and paste your three commands above from an interactive prompt
(see "prompt" CLI command) I have to press two times enter to ensure all
three commands have been taken into an account.
Here is an extract of the documentation about how to send command to the
haproxy CLI:
9.3. Unix Socket commands
The non-interactive mode is the default when socat connects to the
socket. In this mode, a single line may be sent. It is processed as a
whole, responses are sent back, and the connection closes after the end
of the response. This is the mode that scripts and monitoring tools use.
It is possible to send multiple commands in this mode, they need to be
delimited by a semi-colon (';'). For
example :
# echo "show info;show stat;show table" | socat /var/run/haproxy stdio
so this command:
$ echo "trace quic sink buf0; trace quic level developer; trace quic
verbosity clean; trace quic start now; trace qmux sink buf0; trace qmux
level developer; trace qmux verbosity minimal; trace qmux start now;
trace stream sink buf0; trace stream level developer; trace stream
verbosity clean; trace stream start now" | socat stdio
should work (tested after haven copied and pasted from this mail).
Re: [EXTERNAL] Re: HTTP/3 -- POST requests not working
On 4/12/22 14:07, Frederic Lecaille wrote: Please, you could you double check on your side the "stream" traces are correctly enabled? Also ensure you provide use with traces dumped by haproxy when you validate the PHP form. These are the trace commands that I am sending to the stats socket: trace quic sink buf0; trace quic level developer; trace quic verbosity clean; trace quic start now trace qmux sink buf0; trace qmux level developer; trace qmux verbosity minimal; trace qmux start now trace stream sink buf0; trace stream level developer; trace stream verbosity clean; trace stream start now I am sure that I have pressed enter after the last line. If I have copied that third line correctly, then nothing is being logged for stream. The "index.php" file is not equipped to deal with a POST request, but apache doesn't log anything, so I don't think it's ever getting to PHP. It correctly executes the script when using http2. The script just checks a header that I am setting in haproxy, to say "YES" or "NO" depending on whether the browser was doing http3. https://http3test.elyograg.org/ If you load the page in Firefox, you need to do a regular page reload to get it doing http3. In Chrome, it needs to be a shift-reload. If you do vice-versa, it will never switch to http3. I wonder which behavior could be considered a bug. I expect that there is probably no set release data for 2.6.0, but is there a ballpark date when it MIGHT be ready for release? Thanks, Shawn
Re: [EXTERNAL] Re: HTTP/3 -- POST requests not working
Hello Shawn, On 4/12/22 18:30, Shawn Heisey wrote: > On 4/12/22 09:45, Amaury Denoyelle wrote: >> After much analysis of the code, it may be useful to have a run with the >> stream traces as well : >> $ trace stream sink buf0; trace stream level developer; trace stream >> verbosity clean; trace stream start now > > All 3 traces enabled, this time it should only be for one connection, I > think the last one there were two: It's ok for the quic and qmux parts. But I do not seen any trace for the stream part. You should see such a traces when your h3 client open a stream (truncated here, without timestamp). Note the second field of the traces: "quic", "qmux", "stream" [04|quic|2|t_quic.c:2543] RX frame : qc@0x7f56cc3c08e0 STREAM_A uni=1 fin=0 id=10 off=0 len=1 [04|qmux|5|ux_quic.c:101] qcs_new(): entering : qcc=0x7f56cc376e60(F) [04|qmux|5|ux_quic.c:144] qcs_new(): leaving : qcc=0x7f56cc376e60(F) qcs=0x7f56cc35cf20(6) [04|qmux|5|ux_quic.c:101] qcs_new(): entering : qcc=0x7f56cc376e60(F) [04|qmux|5|ux_quic.c:144] qcs_new(): leaving : qcc=0x7f56cc376e60(F) qcs=0x7f56cc35af20(10) . . . [04|stream|5|/stream.c:348] stream_new(): entering [04|stream|5|/stream.c:566] stream_new(): leaving : [2,HTX] SI=(EST,INI) . . . Please, you could you double check on your side the "stream" traces are correctly enabled? Also ensure you provide use with traces dumped by haproxy when you validate the PHP form. Thank you.
Re: HTTP/3 -- POST requests not working
On 4/12/22 09:45, Amaury Denoyelle wrote: After much analysis of the code, it may be useful to have a run with the stream traces as well : $ trace stream sink buf0; trace stream level developer; trace stream verbosity clean; trace stream start now All 3 traces enabled, this time it should only be for one connection, I think the last one there were two: https://paste.elyograg.org/view/a06556ef You can also provide us your haproxy config file, just remove any sensitive info if you have any. https://paste.elyograg.org/view/3f884d8f The acl lines with redacted info are not being matched for the connections that don't work, so it should have no connection to this problem. Thanks, Shawn
Re: HTTP/3 -- POST requests not working
On Tue, Apr 12, 2022 at 05:26:41PM +0200, Amaury Denoyelle wrote: > On Tue, Apr 12, 2022 at 08:01:59AM -0600, Shawn Heisey wrote: > > On 4/12/22 02:22, Amaury Denoyelle wrote: > > > then you can display the traces with the following command : > > > $ show events buf0 > > > > For the h3 layer, the trace mechanism is not currently implemented. You > > > should instead recompile your haproxy binary with the DEBUG options : > > > -DDEBUG_H3 -DDEBUG_QPACK > > > and watch for the stderr output for your process. > > I got some traces. URL below for accessing it. There's a LOT of data. > > https://paste.elyograg.org/view/8678c875 > > I didn't do the H3 debug yet, if you still need it after looking at the > > traces, let me know. You'll also need to tell me how to make those debugs > > active when I build haproxy. > We analyzed the output of the traces but there is nothing of particular > interest here. What may be useful is to activate the H3 traces. > However, I notice that the -dev5 does not compile with H3 traces > activated. So first, you have to apply the attached patch before > recompiling. To do this, save the attachment and execute the following > command in your source folder : > $ patch -p1 < 0001-BUG-MINOR-h3-fix-build-with-DEBUG_H3.patch > Alternatively, you can also use the latest master with contains the > mentionned patch. However, I think it's safer to stay on the -dev5 tag > for the moment. > Once the patch is applied, recompile your haproxy binary with DEBUG_H3 > and DEBUG_QPACK flags. Here is a make example, complete it with your > usual set of flags : > $ make TARGET=linux-glibc USE_QUIC=1 DEBUG="-DDEBUG_H3 -DDEBUG_QPACK" ... > You can now restart your process and watch the stderr output and report > it to us. > Again, thank you very much for your help, Hi again, After much analysis of the code, it may be useful to have a run with the stream traces as well : $ trace stream sink buf0; trace stream level developer; trace stream verbosity clean; trace stream start now You can also provide us your haproxy config file, just remove any sensitive info if you have any. Thanks, -- Amaury Denoyelle
Re: HTTP/3 -- POST requests not working
On 4/12/22 08:01, Shawn Heisey wrote: I didn't do the H3 debug yet, if you still need it after looking at > the traces, let me know. You'll also need to tell me how to make > those debugs active when I build haproxy. I figured this out. I put those debug definitions in a DEBUG="stuff" parameter on the build. The 2.6-dev5 build fails. And it looks like this is known ... a change was pushed a short time ago that says it fixes building with DEBUG_H3. Building from the github repo worked and now my version is 2.6-dev5-bb9704-38. I redirected stdout and stderr to files with my systemd service definition. Here's what got sent to stderr: https://paste.elyograg.org/view/ebe7a659 Thanks, Shawn
Re: HTTP/3 -- POST requests not working
On Tue, Apr 12, 2022 at 08:01:59AM -0600, Shawn Heisey wrote: > On 4/12/22 02:22, Amaury Denoyelle wrote: > > then you can display the traces with the following command : > > $ show events buf0 > > > For the h3 layer, the trace mechanism is not currently implemented. You > > should instead recompile your haproxy binary with the DEBUG options : > > -DDEBUG_H3 -DDEBUG_QPACK > > and watch for the stderr output for your process. > I got some traces. URL below for accessing it. There's a LOT of data. > https://paste.elyograg.org/view/8678c875 > I didn't do the H3 debug yet, if you still need it after looking at the > traces, let me know. You'll also need to tell me how to make those debugs > active when I build haproxy. We analyzed the output of the traces but there is nothing of particular interest here. What may be useful is to activate the H3 traces. However, I notice that the -dev5 does not compile with H3 traces activated. So first, you have to apply the attached patch before recompiling. To do this, save the attachment and execute the following command in your source folder : $ patch -p1 < 0001-BUG-MINOR-h3-fix-build-with-DEBUG_H3.patch Alternatively, you can also use the latest master with contains the mentionned patch. However, I think it's safer to stay on the -dev5 tag for the moment. Once the patch is applied, recompile your haproxy binary with DEBUG_H3 and DEBUG_QPACK flags. Here is a make example, complete it with your usual set of flags : $ make TARGET=linux-glibc USE_QUIC=1 DEBUG="-DDEBUG_H3 -DDEBUG_QPACK" ... You can now restart your process and watch the stderr output and report it to us. Again, thank you very much for your help, -- Amaury Denoyelle >From bb970422546c0db08b244d5e5a12a795a605dd64 Mon Sep 17 00:00:00 2001 From: Amaury Denoyelle Date: Tue, 12 Apr 2022 16:40:52 +0200 Subject: [PATCH] BUG/MINOR: h3: fix build with DEBUG_H3 qcs by_id field has been replaced by a new field named "id". Adjust the h3_debug_printf traces. This is the case since the introduction of the qc_stream_desc type. --- src/h3.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/h3.c b/src/h3.c index e39c54135..2d450d8b8 100644 --- a/src/h3.c +++ b/src/h3.c @@ -231,7 +231,7 @@ static int h3_decode_qcs(struct qcs *qcs, int fin, void *ctx) struct buffer *rxbuf = &qcs->rx.buf; int ret; - h3_debug_printf(stderr, "%s: STREAM ID: %llu\n", __func__, qcs->by_id.key); + h3_debug_printf(stderr, "%s: STREAM ID: %lu\n", __func__, qcs->id); if (!b_data(rxbuf)) return 0; @@ -333,7 +333,7 @@ static int h3_control_recv(struct h3_uqs *h3_uqs, void *ctx) struct buffer *rxbuf = &h3_uqs->qcs->rx.buf; struct h3 *h3 = ctx; - h3_debug_printf(stderr, "%s STREAM ID: %llu\n", __func__, h3_uqs->qcs->by_id.key); + h3_debug_printf(stderr, "%s STREAM ID: %lu\n", __func__, h3_uqs->qcs->id); if (!b_data(rxbuf)) return 1; -- 2.35.1
Re: HTTP/3 -- POST requests not working
On 4/12/22 02:22, Amaury Denoyelle wrote: then you can display the traces with the following command : $ show events buf0 For the h3 layer, the trace mechanism is not currently implemented. You should instead recompile your haproxy binary with the DEBUG options : -DDEBUG_H3 -DDEBUG_QPACK and watch for the stderr output for your process. I got some traces. URL below for accessing it. There's a LOT of data. https://paste.elyograg.org/view/8678c875 I didn't do the H3 debug yet, if you still need it after looking at the traces, let me know. You'll also need to tell me how to make those debugs active when I build haproxy. Thanks, Shawn
Re: HTTP/3 -- POST requests not working
On Tue, Apr 12, 2022 at 08:14:25AM +0200, Willy Tarreau wrote: > Hi Shawn, > [...] > On Mon, Apr 11, 2022 at 01:05:21PM -0600, Shawn Heisey wrote: > > An astute observer will notice that the backend says 81 while the packet > > capture shows port 82. This is because I changed the port number on the > > server to point to nginx instead of apache, but did not rename the backend. > > My use of nginx is temporary, only using it to eliminate Apache as the > > problem. I know how to configure Apache; it would take me forever to fully > > switch to nginx. > I think that the QUIC team will soon ask you to enable traces on the QUIC > layer, but I'll let them follow up ;-) Hi Shawn, First, thank you for your interest for QUIC in haproxy :) I have quickly tested POST on my side but did not see any issue. To help you diagnostic your problem, you can try to activate the traces on the stats socket. for the transport traces : $ trace quic sink buf0; trace quic level developer; trace quic verbosity clean; trace quic start now and for the mux : $ trace qmux sink buf0; trace qmux level developer; trace qmux verbosity minimal; trace qmux start now then you can display the traces with the following command : $ show events buf0 For the h3 layer, the trace mechanism is not currently implemented. You should instead recompile your haproxy binary with the DEBUG options : -DDEBUG_H3 -DDEBUG_QPACK and watch for the stderr output for your process. Thanks again for your help, Regards, -- Amaury Denoyelle
Pomoc finansowa w inwestowaniu
Dzień dobry, czy myśleli Państwo o wnioskowaniu w sprawie pozyskania dodatkowych środków na bieżącą działalność i inwestycje? Jeżeli są Państwo zainteresowani wsparciem z tego zakresu, chętnie przedstawię szereg propozycji, zgodnych z Państwa potrzebami i oczekiwaniami. Bardzo proszę o informację czy chcieliby Państwo ubiegać się o zwrotne finansowanie. Przedstawię szczegóły. Pozdrawiam Maciej Dąbrowski
