Re: [EXTERNAL] testing websockets

[Илья Шипицин]

Proposed testsuite only catches issue 737 regression in https mode.

Also, it tests h2, because native chromium is involved.

I'll check whether VTC is able to catch issue 737.

As for proposed test suite, it is really heavy to run on CI. But it might
be used as release quality gate

On Wed, Oct 20, 2021, 9:50 AM Amaury Denoyelle 
wrote:

> On Wed, Oct 20, 2021 at 09:59:59AM +0500, Илья Шипицин wrote:
> > Hello,
> > I've found a way how to test websockets automatically.
> > this approach is able to catch
> https://github.com/haproxy/haproxy/issues/737
> >
> > the idea is to place haproxy between browser and kestrel in SignalR
> tests (
> > https://github.com/dotnet/aspnetcore/tree/main/src/SignalR )
> > test takes 3 hours on my low end laptop.
> > I collected all required parts in
> > https://github.com/chipitsine/haproxy-signalr-tests
> > also, I checked current maintained branches, no issues (last issue was
> > fixed in issue 737).
>
> Hi,
>
> I did not look precisely at your testsuite, but note that now we have
> some websockets related VTC regtests. Of course, this does only test the
> tunnel establishment without websocket data exchange, but it's still a
> good start. Also this will be soon extended with a new test to better
> handle h2 websocket.
>
> Regards,
>
> --
> Amaury Denoyelle
>

Re: [EXTERNAL] testing websockets

[Amaury Denoyelle]

On Wed, Oct 20, 2021 at 09:59:59AM +0500, Илья Шипицин wrote:
> Hello,
> I've found a way how to test websockets automatically.
> this approach is able to catch https://github.com/haproxy/haproxy/issues/737
> 
> the idea is to place haproxy between browser and kestrel in SignalR tests (
> https://github.com/dotnet/aspnetcore/tree/main/src/SignalR )
> test takes 3 hours on my low end laptop.
> I collected all required parts in
> https://github.com/chipitsine/haproxy-signalr-tests
> also, I checked current maintained branches, no issues (last issue was
> fixed in issue 737).

Hi,

I did not look precisely at your testsuite, but note that now we have
some websockets related VTC regtests. Of course, this does only test the
tunnel establishment without websocket data exchange, but it's still a
good start. Also this will be soon extended with a new test to better
handle h2 websocket.

Regards,

-- 
Amaury Denoyelle

testing websockets

[Илья Шипицин]

Hello,

I've found a way how to test websockets automatically.
this approach is able to catch https://github.com/haproxy/haproxy/issues/737


the idea is to place haproxy between browser and kestrel in SignalR tests (
https://github.com/dotnet/aspnetcore/tree/main/src/SignalR )

test takes 3 hours on my low end laptop.

I collected all required parts in
https://github.com/chipitsine/haproxy-signalr-tests
also, I checked current maintained branches, no issues (last issue was
fixed in issue 737).

cheers,
Ilya

Re: [PATCH v2] BUILD: SSL: function "ERR_func_error_string" is deprecated in OpenSSL-3.0.0

[Илья Шипицин]

similar patchset

https://patchwork.openvpn.net/project/openvpn2/list/?series=1309

Willy, please forward to SSL support team

чт, 7 окт. 2021 г. в 14:08, Илья Шипицин :

>
>
> чт, 7 окт. 2021 г. в 12:49, Willy Tarreau :
>
>> On Thu, Oct 07, 2021 at 11:30:54AM +0500,  ??? wrote:
>> > > Just thinking about something, given that the new API was already
>> adopted
>> > > by BoringSSL and will probably be at some point in time by LibreSSL,
>> would
>> > > it not be better to have a single macro "HA_SSL_USE_API_V3" or
>> something
>> > > like this that we set based on the various libs' versions, and rely
>> on this
>> > > one for all other defines ? I think it could significantly simplify
>> the
>> > > porting to other libs and avoid a real mess with version numbers
>> > > everywhere.
>> > >
>> >
>> > even BoringSSL states "it adopted upstream changes", it is different in
>> > details, for example ERR_func_error_string
>> > is not deprecated in BoringSSL.
>> >
>> > Well, there might be a common divisor of course. I'll keep an eye on it
>> :)
>> >
>> > as for this particular patch, it is openssl specific (at least now)
>>
>> OK. I'll let the SSL maintainers deal with this.
>>
>
> I set up Fedora Rawhide builds.
> Fedora now uses openssl-3.0.0, all builds are bloody murder:
>
> fedora_clang (#1657157053) · Jobs · Ilya Shipitsin / haproxy-ci-playground
> · GitLab
> 
>
>
>
>>
>> Thanks!
>> Willy
>>
>

Re: compression offload ... in default section

[Björn Jacke]

Hi,

On 19.10.21 11:06, Christopher Faulet wrote:
Sorry Björn, I missed your reply. It is strange, there is no known bug 
in this area for now. There is probably something in the request or 
response headers preventing the compression to be enabled.


I found the error: the "compression offload" is not honored in the 
default section. This behavior is documented but it slipped from my memory.


What is actually the reason why this is silently ignored when it is set 
in the the default section? If someone has a reason to set that in the 
default section, why does haproxy ignore it intentionally?


If the support of compression offload shall stay intentionally 
unsupported in the default section, then it would be good if this would 
trigger a configure check warning, wouldn't it?


Thanks
Björn

Re: PH disconnects, but "show errors" has 0 entries ?

[Christopher Faulet]

Le 10/19/21 à 16:49, Jim Freeman a écrit :

OK - this is weird (so don't shoot the messenger?).
With more tcpdump-ing and examination, the back-end service logs that it sent a 
response, but

  1) tcpdump running on the haproxy instance never sees the response !
      a) 2 proxies - an AWS ELB and on-instance nginx - lie between HAProxy 
instance and the service
  2) sans any response (and within 0.2 to 13 seconds of the request send), 
HAProxy initiates the PH/500 to the client!


It would make sense to me if any timeouts or disconnects were involved - HAProxy 
would report an [sS][DH] or somesuch.


And reverting the sending of the "content-security-policy: frame-ancestors ..." 
and "x-frame-options: ..." response(!) headers makes the problem disappear 
again.  You'll rightly point out that HTTP/1.1 is stateless, and that the prior 
history of the request/response stream (and response headers sent to the client) 
shouldn't affect the (non-)response to a given request.


Any clues as to how/why the PH/500 might be generated without a response to 
trigger it would be most eagerly received.  While it is entirely likely this 
will wind up being a "nut loose on the keyboard" issue, I just thought I'd share 
my observations and befuddlement ...




First of all, I missed a point. The 2.2.8 is quite old. You must upgrade first. 
Then, have you check the rewrite error counters on your backend ? Because, 
AFAIK, it is the only place where a 500 is possible with the PH termination 
state. If you are using http-after-response rules, it may explain this error.


However, share your redacted configuration too. It can help me to explain what 
you observe.


--
Christopher Faulet

Re: PH disconnects, but "show errors" has 0 entries ?

[Jim Freeman]

OK - this is weird (so don't shoot the messenger?).
With more tcpdump-ing and examination, the back-end service logs that it
sent a response, but
 1) tcpdump running on the haproxy instance never sees the response !
 a) 2 proxies - an AWS ELB and on-instance nginx - lie between HAProxy
instance and the service
 2) sans any response (and within 0.2 to 13 seconds of the request send),
HAProxy initiates the PH/500 to the client!

It would make sense to me if any timeouts or disconnects were involved -
HAProxy would report an [sS][DH] or somesuch.

And reverting the sending of the "content-security-policy: frame-ancestors
..." and "x-frame-options: ..." response(!) headers makes the problem
disappear again.  You'll rightly point out that HTTP/1.1 is stateless, and
that the prior history of the request/response stream (and response headers
sent to the client) shouldn't affect the (non-)response to a given request.

Any clues as to how/why the PH/500 might be generated without a response to
trigger it would be most eagerly received.  While it is entirely likely
this will wind up being a "nut loose on the keyboard" issue, I just thought
I'd share my observations and befuddlement ...

https://www.mail-archive.com/haproxy@formilux.org/msg41308.html

"This computer stuff is hard ..."

On Tue, Oct 19, 2021 at 3:24 AM Christopher Faulet 
wrote:

> Le 10/13/21 à 8:30 PM, Jim Freeman a écrit :
> > In adding a couple of new security response headers via haproxy.cfg (one
> is 112
> > bytes, the other 32), a few requests are now getting 500 status (PH
> session
> > state) responses, but "show errors" has 0 entries?  Most responses
> succeed (all
> > have the additional headers), so it's not a problem with the new headers
> themselves.
> >
> > If haproxy generates a PH/500, shouldn't "show errors" show details of
> the
> > offending response ?
> >
> > Thanks,
> > ...jfree
> > ==
> > # echo "show info" | socat stdio /run/haproxy/stats.sock | grep ^Version:
> > Version: 2.2.8-1~bpo10+1
> >
> > #  echo "show errors -1" | socat - /run/haproxy/stats.sock
> > Total events captured on [13/Oct/2021:18:24:15.819] : 0
> >
> > # cat /etc/debian_version
> > 10.11
>
> Hi,
>
> Only parsing errors are reported by "show errors" command. Here PH/500
> error is
> most probably due to a header rewrite error. I have not deeply checked
> however.
> You can verify my assumption by checking the "wrew" counter in "show
> stats"
> command output on the stats socket.
>
> Header rewrite errors are triggered when there is not enough space in the
> buffer
> to perform the rewrites. By default, 1024 Bytes are reserved in the
> buffer, to
> be sure to have enough space to perform some rewrites. If you add many
> headers
> in the response, it may be the problem. You can increase the reserve by
> setting
> "tune.maxrewrite" global parameter.
>
> When such error is encountered, HAProxy returns a 500-Internal-Error
> response.
> You can change that to make it fails silently. To do so, take a look at
> the
> "strict-mode" http-response action.
>
> --
> Christopher Faulet
>

Re: PH disconnects, but "show errors" has 0 entries ?

[Jim Freeman]

Many thanks for your insight and response - I'll check that out.

On Tue, Oct 19, 2021 at 3:24 AM Christopher Faulet 
wrote:

> Le 10/13/21 à 8:30 PM, Jim Freeman a écrit :
> > In adding a couple of new security response headers via haproxy.cfg (one
> is 112
> > bytes, the other 32), a few requests are now getting 500 status (PH
> session
> > state) responses, but "show errors" has 0 entries?  Most responses
> succeed (all
> > have the additional headers), so it's not a problem with the new headers
> themselves.
> >
> > If haproxy generates a PH/500, shouldn't "show errors" show details of
> the
> > offending response ?
> >
> > Thanks,
> > ...jfree
> > ==
> > # echo "show info" | socat stdio /run/haproxy/stats.sock | grep ^Version:
> > Version: 2.2.8-1~bpo10+1
> >
> > #  echo "show errors -1" | socat - /run/haproxy/stats.sock
> > Total events captured on [13/Oct/2021:18:24:15.819] : 0
> >
> > # cat /etc/debian_version
> > 10.11
>
> Hi,
>
> Only parsing errors are reported by "show errors" command. Here PH/500
> error is
> most probably due to a header rewrite error. I have not deeply checked
> however.
> You can verify my assumption by checking the "wrew" counter in "show
> stats"
> command output on the stats socket.
>
> Header rewrite errors are triggered when there is not enough space in the
> buffer
> to perform the rewrites. By default, 1024 Bytes are reserved in the
> buffer, to
> be sure to have enough space to perform some rewrites. If you add many
> headers
> in the response, it may be the problem. You can increase the reserve by
> setting
> "tune.maxrewrite" global parameter.
>
> When such error is encountered, HAProxy returns a 500-Internal-Error
> response.
> You can change that to make it fails silently. To do so, take a look at
> the
> "strict-mode" http-response action.
>
> --
> Christopher Faulet
>

Re: PH disconnects, but "show errors" has 0 entries ?

[Christopher Faulet]

Le 10/13/21 à 8:30 PM, Jim Freeman a écrit :
In adding a couple of new security response headers via haproxy.cfg (one is 112 
bytes, the other 32), a few requests are now getting 500 status (PH session 
state) responses, but "show errors" has 0 entries?  Most responses succeed (all 
have the additional headers), so it's not a problem with the new headers themselves.


If haproxy generates a PH/500, shouldn't "show errors" show details of the 
offending response ?


Thanks,
...jfree
==
# echo "show info" | socat stdio /run/haproxy/stats.sock | grep ^Version:
Version: 2.2.8-1~bpo10+1

#  echo "show errors -1" | socat - /run/haproxy/stats.sock
Total events captured on [13/Oct/2021:18:24:15.819] : 0

# cat /etc/debian_version
10.11


Hi,

Only parsing errors are reported by "show errors" command. Here PH/500 error is 
most probably due to a header rewrite error. I have not deeply checked however. 
You can verify my assumption by checking the "wrew" counter in "show stats" 
command output on the stats socket.


Header rewrite errors are triggered when there is not enough space in the buffer 
to perform the rewrites. By default, 1024 Bytes are reserved in the buffer, to 
be sure to have enough space to perform some rewrites. If you add many headers 
in the response, it may be the problem. You can increase the reserve by setting 
"tune.maxrewrite" global parameter.


When such error is encountered, HAProxy returns a 500-Internal-Error response. 
You can change that to make it fails silently. To do so, take a look at the 
"strict-mode" http-response action.


--
Christopher Faulet

Re: compression offload and http2

[Christopher Faulet]

Le 10/15/21 à 12:04 PM, Björn Jacke a écrit :

On 15.10.21 10:10, Christopher Faulet wrote:


It should work. What is your HAProxy version ?


2.4.7

Björn



Sorry Björn, I missed your reply. It is strange, there is no known bug in this 
area for now. There is probably something in the request or response headers 
preventing the compression to be enabled. If it is easy to reproduce on a test 
platform, you may try to start HAproxy in foreground in debug mode (-db -d on 
the command line). This will print the request and the response on stderr.


--
Christopher Faulet

Re: [PATCH] BUILD: improve reproducibility by filtering BUILD_CFLAGS

[Vincent Bernat]

 ❦ 19 October 2021 09:22 +02, Vincent Bernat:

> This could be backported to 2.4. Older versions do not display CFLAGS.

Note that if you find this too ugly, I have no problem to maintain this
as an OOT patch.
-- 
Avoid unnecessary branches.
- The Elements of Programming Style (Kernighan & Plauger)

[PATCH] BUILD: improve reproducibility by filtering BUILD_CFLAGS

[Vincent Bernat]

Some distributions (Debian) adds `-ffile-prefix-map=/current/pwd=` to
CFLAGS in an attempt to make the package more reproducible when source
code is using `__FILE__`. Unfortunately, this makes HAProxy build not
reproducible since CFLAGS is recorded to be displayed in `haproxy
--version`. To solve this, we filter out these arguments as they are
not really important to report to the user.

This could be backported to 2.4. Older versions do not display CFLAGS.
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 50a21b3105cd..6597f0bf855f 100644
--- a/Makefile
+++ b/Makefile
@@ -968,7 +968,7 @@ src/haproxy.o:  src/haproxy.c $(DEP)
  -DBUILD_ARCH='"$(strip $(ARCH))"' \
  -DBUILD_CPU='"$(strip $(CPU))"' \
  -DBUILD_CC='"$(strip $(CC))"' \
- -DBUILD_CFLAGS='"$(strip $(VERBOSE_CFLAGS))"' \
+ -DBUILD_CFLAGS='"$(filter-out -ffile-prefix-map=% 
-fmacro-prefix-map=% -fdebug-prefix-map=%,$(strip $(VERBOSE_CFLAGS)))"' \
  -DBUILD_OPTIONS='"$(strip $(BUILD_OPTIONS))"' \
  -DBUILD_DEBUG='"$(strip $(DEBUG))"' \
  -DBUILD_FEATURES='"$(strip $(BUILD_FEATURES))"' \
-- 
2.33.0