Re: Issue with HTX
Le 16/10/2019 à 08:48, GARDAIS Ionel a écrit : Hi Christopher, First : good news : I was not able to reproduce the issue with 2.1-dev2-e0f48a-88 and HTX enable. I guess e0f8dc576f62ace9ad1055ca068ab5d4f3a952aa was the culprit. To answer your questions and for others on the list : - The issue arose with H1 (H2 not enable) - It's the client who complained about a malformed payload thus it was unable to unmarshall datas into a valid object (it was the result of an API call). The message from the client is a short error message like "expecting [0-9a-fA-F] but got 0x4f" (note that 0x4f is not a constant and would vary from run to run) - To my understanding, the payload is chunk-encoded. Hi, Thanks to confirm. So it is indeed a problem of chunking fixed by the commit e0f8dc576. The patch was backported to 2.0 and 1.9. Thanks for your help, -- Christopher Faulet
Re: Issue with HTX
Hi Christopher, First : good news : I was not able to reproduce the issue with 2.1-dev2-e0f48a-88 and HTX enable. I guess e0f8dc576f62ace9ad1055ca068ab5d4f3a952aa was the culprit. To answer your questions and for others on the list : - The issue arose with H1 (H2 not enable) - It's the client who complained about a malformed payload thus it was unable to unmarshall datas into a valid object (it was the result of an API call). The message from the client is a short error message like "expecting [0-9a-fA-F] but got 0x4f" (note that 0x4f is not a constant and would vary from run to run) - To my understanding, the payload is chunk-encoded. -- Ionel GARDAIS Tech'Advantage CIO - IT Team manager - Mail original - De: "Christopher Faulet" À: "Ionel GARDAIS" , "Willy Tarreau" Cc: "haproxy" Envoyé: Mardi 15 Octobre 2019 10:20:50 Objet: Re: Issue with HTX Le 12/10/2019 à 14:23, GARDAIS Ionel a écrit : > I might have been too enthusiast. > haproxy is set as a TLS-endpoint thus the traces from the client-side are > encrypted. > The trace of the server side is plain text but as this is the client which > complains about a malformed packet, client-side should be compared to > server-side. > > Instead of a tcpdump, I should run full debug on haproxy. > Could you tell me how to do this for HTX ? > > Also, as I was looking at the traces, note that protobuf is involved in the > client-server exchange. > It might change how debug and traces are collected. > Hi, The data corruption happens for H1 or H2 connections (or both) ? On the client side or the server side ? If it is related to H1 connections, is the payload chunk-encoded or not ? If yes, I pushed a fix yesterday that may help. If so, give the last 2.1 snapshot a try. Then, which kind of data corruption did you observe and how did you observe it ? Finally, could you share your configuration please ? Thanks, -- Christopher Faulet -- 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
Re: Issue with HTX
Le 12/10/2019 à 14:23, GARDAIS Ionel a écrit : I might have been too enthusiast. haproxy is set as a TLS-endpoint thus the traces from the client-side are encrypted. The trace of the server side is plain text but as this is the client which complains about a malformed packet, client-side should be compared to server-side. Instead of a tcpdump, I should run full debug on haproxy. Could you tell me how to do this for HTX ? Also, as I was looking at the traces, note that protobuf is involved in the client-server exchange. It might change how debug and traces are collected. Hi, The data corruption happens for H1 or H2 connections (or both) ? On the client side or the server side ? If it is related to H1 connections, is the payload chunk-encoded or not ? If yes, I pushed a fix yesterday that may help. If so, give the last 2.1 snapshot a try. Then, which kind of data corruption did you observe and how did you observe it ? Finally, could you share your configuration please ? Thanks, -- Christopher Faulet
Re: Issue with HTX
I might have been too enthusiast. haproxy is set as a TLS-endpoint thus the traces from the client-side are encrypted. The trace of the server side is plain text but as this is the client which complains about a malformed packet, client-side should be compared to server-side. Instead of a tcpdump, I should run full debug on haproxy. Could you tell me how to do this for HTX ? Also, as I was looking at the traces, note that protobuf is involved in the client-server exchange. It might change how debug and traces are collected. -- Ionel GARDAIS Tech'Advantage CIO - IT Team manager - Mail original - De: "Willy Tarreau" À: "Ionel GARDAIS" Cc: "haproxy" Envoyé: Samedi 12 Octobre 2019 06:20:40 Objet: Re: Issue with HTX Hi Ionel, On Fri, Oct 11, 2019 at 10:49:19PM +0200, GARDAIS Ionel wrote: > Hi Willy, > > Got 2 tcpdump but I really don't know where to start searching. > One is with HTX on the other HTX disabled. Perfect! > May I send them to you ? Yes please. However I won't have time to look at them before ~Tuesday. Thanks! Willy -- 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
Re: Issue with HTX
Hi Ionel, On Fri, Oct 11, 2019 at 10:49:19PM +0200, GARDAIS Ionel wrote: > Hi Willy, > > Got 2 tcpdump but I really don't know where to start searching. > One is with HTX on the other HTX disabled. Perfect! > May I send them to you ? Yes please. However I won't have time to look at them before ~Tuesday. Thanks! Willy
Re: Issue with HTX
Hi Willy, Got 2 tcpdump but I really don't know where to start searching. One is with HTX on the other HTX disabled. May I send them to you ? -- Ionel GARDAIS Tech'Advantage CIO - IT Team manager -- 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
Re: Issue with HTX
> Not for now, this requires the trace filters which are only planned but > not yet available. Thus you'll have debugging enabled on the whole process > at once. Can your reproduce this outside production ? No. But I can run tcpdump filtered on the backend's server IP and provide you with a timestamp of when the client triggers the error because of malformed datas. Ionel -- 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
Re: Issue with HTX
On Wed, Oct 09, 2019 at 05:58:31PM +0200, GARDAIS Ionel wrote: > Was using 2.0.7. > Had the same issue with 2.1-dev2 > > It happens with the same tool but does not seem to be triggered every time. OK this is already a good thing. > Is it possible to enable H1+H2 debug traces on a specific backend ? Not for now, this requires the trace filters which are only planned but not yet available. Thus you'll have debugging enabled on the whole process at once. Can your reproduce this outside production ? Willy
Re: Issue with HTX
Was using 2.0.7. Had the same issue with 2.1-dev2 It happens with the same tool but does not seem to be triggered every time. Is it possible to enable H1+H2 debug traces on a specific backend ? -- Ionel GARDAIS Tech'Advantage CIO - IT Team manager - Mail original - De: "Willy Tarreau" À: "Ionel GARDAIS" Cc: "haproxy" Envoyé: Mercredi 9 Octobre 2019 04:27:42 Objet: Re: Issue with HTX Hi Ionel, On Tue, Oct 08, 2019 at 03:33:16PM +0200, GARDAIS Ionel wrote: > Hi, > > I'm facing a data corruption that seems related to HTX. > Simply adding 'no option http-use-htx' solve the issue. > > What kind of traces do you need to help diagnose the issue and how to collect > it ? Ideally a network capture before and after haproxy will help. Are you sure you're running on an up-to-date version ? Christopher addressed one such issue recently in H2. If you're able to reproduce this at will, it may be useful to run on latest 2.1-dev, where we can help you enable H1+H2 debug traces. Thanks, Willy -- 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
Re: Issue with HTX
Hi Ionel, On Tue, Oct 08, 2019 at 03:33:16PM +0200, GARDAIS Ionel wrote: > Hi, > > I'm facing a data corruption that seems related to HTX. > Simply adding 'no option http-use-htx' solve the issue. > > What kind of traces do you need to help diagnose the issue and how to collect > it ? Ideally a network capture before and after haproxy will help. Are you sure you're running on an up-to-date version ? Christopher addressed one such issue recently in H2. If you're able to reproduce this at will, it may be useful to run on latest 2.1-dev, where we can help you enable H1+H2 debug traces. Thanks, Willy
Issue with HTX
Hi, I'm facing a data corruption that seems related to HTX. Simply adding 'no option http-use-htx' solve the issue. What kind of traces do you need to help diagnose the issue and how to collect it ? Thanks, Ionel -- 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
