Re: FCGI calls return 500 with "IH" Stream State
Hi.
I have added fcgi trace
```
global
log stdout format raw daemon debug
pidfile /data/haproxy/run/haproxy.pid
# maxconn auto config from hap
# nbthread auto config from hap
master-worker
#tune.comp.maxlevel 5
expose-experimental-directives
trace fcgi sink stdout
trace fcgi verbosity advanced
trace fcgi event any
trace fcgi start now
# turn on stats unix socket
stats socket /data/haproxy/run/stats mode 660 level admin expose-fd listeners
```
and created with that output a issue.
https://github.com/haproxy/haproxy/issues/2568
Regards
Alex
On 2024-05-16 (Do.) 17:05, Aleksandar Lazic wrote:
Hi.
I have a strange behavior with HAProxy and FCGI PHP App.
When I call an admin URL returns HAProxy a 500, after a refresh of the same
page returns the HAProxy 200.
```
10.128.2.35:39684 [16/May/2024:14:54:26.229] craft-cms fcgi-servers/craftcms1
0/0/0/-1/1138 500 15416 - - IH-- 2/2/0/0/0 0/0 "GET /craftcms/admin/settings
HTTP/1.1"
10.131.0.26:46546 [16/May/2024:14:56:01.870] craft-cms fcgi-servers/craftcms1
0/0/0/1511/1514 200 113460 - - 2/2/0/0/0 0/0 "GET
/craftcms/admin/settings HTTP/1.1"
```
How can I debug this 'I' flag which should never happen as the doc say.
https://docs.haproxy.org/2.9/configuration.html#8.5
```
I : an internal error was identified by the proxy during a self-check.
This should NEVER happen, and you are encouraged to report any log
containing this, because this would almost certainly be a bug. It
would be wise to preventively restart the process after such an
event too, in case it would be caused by memory corruption.
```
I use the latest haproxy image haproxytech/haproxy-ubuntu:2.9 in OpenShift
with that config.
```
global
log stdout format raw daemon debug
pidfile /data/haproxy/run/haproxy.pid
# maxconn auto config from hap
# nbthread auto config from hap
master-worker
tune.comp.maxlevel 5
# turn on stats unix socket
stats socket /data/haproxy/run/stats mode 660 level admin expose-fd
listeners
resolvers kube-dns
nameserver dns1 dns-default.openshift-dns.svc.cluster.local:53
accepted_payload_size 4096
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold other 30s
hold refused 30s
hold nx 30s
hold timeout 30s
hold valid 10s
hold obsolete 30s
defaults
mode http
balance leastconn
log global
option httplog
option dontlognull
option log-health-checks
option forwardfor except 10.196.106.108/32
option redispatch
retries 3
timeout http-request 10s
timeout queue 30s
timeout connect 10s
timeout client 30s
timeout server 30s
timeout http-keep-alive 10s
timeout check 10s
#maxconn 3000
frontend craft-cms
bind *:8080
tcp-request inspect-delay 5s
tcp-request content accept if HTTP
# default check url from appgateway
monitor-uri /health
# https://www.haproxy.com/blog/load-balancing-php-fpm-with-haproxy-and-fastcgi
# fix CVE-2019-11043
http-request deny if { path_sub -i %0a %0d }
# Mitigate CVE-2023-40225 (Proxy forwards malformed empty Content-Length
headers)
http-request deny if { hdr_len(content-length) 0 }
# Strip off Proxy headers to prevent HTTpoxy (https://httpoxy.org/)
http-request del-header Proxy
# DNS labels are case insensitive (RFC 4343), we need to convert the
hostname into lowercase
# before matching, or any requests containing uppercase characters will
never match.
http-request set-header Host %[req.hdr(Host),lower]
acl exist-php-ext path_sub -i .php
acl fpm-status path /fpm-status
http-request set-path /index.php%[path] if !exist-php-ext !fpm-status !{
path_end .php }
# https://www.haproxy.com/blog/haproxy-and-http-strict-transport-security-hsts
# max-age is mandatory
# 1600 seconds is a bit more than 6 months
http-response set-header Strict-Transport-Security "max-age=1600;
includeSubDomains; preload;"
default_backend fcgi-servers
listen stats
bind *:1936
# Health check monitoring uri.
monitor-uri /healthz
# provide prometheus endpoint
http-request use-service prometheus-exporter if { path /metrics }
# Add your custom health check monitoring failure condition here.
# monitor fail if
stats enable
stats uri /
backend fcgi-servers
option httpchk
http-check connect proto fcgi
http-check send meth GET uri /fpm-ping
use-fcgi-app php-fpm
# https://www.haproxy.com/blog/circuit-breaking-haproxy
server-template craftcms 10
"${CRAFT_SERVICE}.${NAMESPACE}.svc.cluster.local":9000 proto fcgi check
resolvers kube-dns init-addr none
FCGI calls return 500 with "IH" Stream State
Hi.
I have a strange behavior with HAProxy and FCGI PHP App.
When I call an admin URL returns HAProxy a 500, after a refresh of the same page
returns the HAProxy 200.
```
10.128.2.35:39684 [16/May/2024:14:54:26.229] craft-cms fcgi-servers/craftcms1
0/0/0/-1/1138 500 15416 - - IH-- 2/2/0/0/0 0/0 "GET /craftcms/admin/settings
HTTP/1.1"
10.131.0.26:46546 [16/May/2024:14:56:01.870] craft-cms fcgi-servers/craftcms1
0/0/0/1511/1514 200 113460 - - 2/2/0/0/0 0/0 "GET /craftcms/admin/settings
HTTP/1.1"
```
How can I debug this 'I' flag which should never happen as the doc say.
https://docs.haproxy.org/2.9/configuration.html#8.5
```
I : an internal error was identified by the proxy during a self-check.
This should NEVER happen, and you are encouraged to report any log
containing this, because this would almost certainly be a bug. It
would be wise to preventively restart the process after such an
event too, in case it would be caused by memory corruption.
```
I use the latest haproxy image haproxytech/haproxy-ubuntu:2.9 in OpenShift with
that config.
```
global
log stdout format raw daemon debug
pidfile /data/haproxy/run/haproxy.pid
# maxconn auto config from hap
# nbthread auto config from hap
master-worker
tune.comp.maxlevel 5
# turn on stats unix socket
stats socket /data/haproxy/run/stats mode 660 level admin expose-fd
listeners
resolvers kube-dns
nameserver dns1 dns-default.openshift-dns.svc.cluster.local:53
accepted_payload_size 4096
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold other 30s
hold refused 30s
hold nx 30s
hold timeout 30s
hold valid 10s
hold obsolete30s
defaults
modehttp
balance leastconn
log global
option httplog
option dontlognull
option log-health-checks
option forwardfor except 10.196.106.108/32
option redispatch
retries 3
timeout http-request10s
timeout queue 30s
timeout connect 10s
timeout client 30s
timeout server 30s
timeout http-keep-alive 10s
timeout check 10s
#maxconn 3000
frontend craft-cms
bind *:8080
tcp-request inspect-delay 5s
tcp-request content accept if HTTP
# default check url from appgateway
monitor-uri /health
# https://www.haproxy.com/blog/load-balancing-php-fpm-with-haproxy-and-fastcgi
# fix CVE-2019-11043
http-request deny if { path_sub -i %0a %0d }
# Mitigate CVE-2023-40225 (Proxy forwards malformed empty Content-Length
headers)
http-request deny if { hdr_len(content-length) 0 }
# Strip off Proxy headers to prevent HTTpoxy (https://httpoxy.org/)
http-request del-header Proxy
# DNS labels are case insensitive (RFC 4343), we need to convert the hostname
into lowercase
# before matching, or any requests containing uppercase characters will never
match.
http-request set-header Host %[req.hdr(Host),lower]
acl exist-php-ext path_sub -i .php
acl fpm-status path /fpm-status
http-request set-path /index.php%[path] if !exist-php-ext !fpm-status !{
path_end .php }
# https://www.haproxy.com/blog/haproxy-and-http-strict-transport-security-hsts
# max-age is mandatory
# 1600 seconds is a bit more than 6 months
http-response set-header Strict-Transport-Security "max-age=1600;
includeSubDomains; preload;"
default_backend fcgi-servers
listen stats
bind *:1936
# Health check monitoring uri.
monitor-uri /healthz
# provide prometheus endpoint
http-request use-service prometheus-exporter if { path /metrics }
# Add your custom health check monitoring failure condition here.
# monitor fail if
stats enable
stats uri /
backend fcgi-servers
option httpchk
http-check connect proto fcgi
http-check send meth GET uri /fpm-ping
use-fcgi-app php-fpm
# https://www.haproxy.com/blog/circuit-breaking-haproxy
server-template craftcms 10
"${CRAFT_SERVICE}.${NAMESPACE}.svc.cluster.local":9000 proto fcgi check
resolvers kube-dns init-addr none observe layer7 error-limit 5 on-error
mark-down inter 10s rise 30 slowstart 40s
fcgi-app php-fpm
log-stderr global
option keep-conn
option mpxs-conns
option max-reqs 10
docroot /app/web
index index.php
path-info ^(/.+\.php)(/.*)?$
```
[PATCH v2] FEATURE: add opt-in MPTCP support
From: Dorian Craps
Multipath TCP (MPTCP), standardized in RFC8684 [1], is a TCP extension
that enables a TCP connection to use different paths.
Multipath TCP has been used for several use cases. On smartphones, MPTCP
enables seamless handovers between cellular and Wi-Fi networks while
preserving established connections. This use-case is what pushed Apple
to use MPTCP since 2013 in multiple applications [2]. On dual-stack
hosts, Multipath TCP enables the TCP connection to automatically use the
best performing path, either IPv4 or IPv6. If one path fails, MPTCP
automatically uses the other path.
To benefit from MPTCP, both the client and the server have to support
it. Multipath TCP is a backward-compatible TCP extension that is enabled
by default on recent Linux distributions (Debian, Ubuntu, Redhat, ...).
Multipath TCP is included in the Linux kernel since version 5.6 [3]. To
use it on Linux, an application must explicitly enable it when creating
the socket. No need to change anything else in the application.
This attached patch adds the "mptcp" global option in the config, which
allows the creation of an MPTCP socket instead of TCP on Linux. If
Multipath TCP is not supported on the system, an error will be reported,
and the application will stop.
A test has been added, it is a copy of "default_rules.vtc" in tcp-rules
with the addition of "mptcp" in the config. I'm not sure what else needs
to be tested for the moment, with this global MPTCP option.
Note: another patch is coming to support enabling MPTCP per address, but
I prefer to already send this patch, just in case, as I will soon have
less time to dedicate to this.
Due to the limited impact within a data center environment,
we have decided not to implement MPTCP between the proxy and the servers.
The high-speed, low-latency nature of data center networks reduces
the benefits of MPTCP, making the complexity of its implementation
unnecessary in this context.
Developed with the help of Matthieu Baerts (matt...@kernel.org) and
Olivier Bonaventure (olivier.bonavent...@uclouvain.be)
Link: https://www.rfc-editor.org/rfc/rfc8684.html [1]
Link: https://www.tessares.net/apples-mptcp-story-so-far/ [2]
Link: https://www.mptcp.dev [3]
---
doc/configuration.txt | 9 +++
include/haproxy/proto_rhttp.h | 2 +
reg-tests/tcp-rules/default_rules_mptcp.vtc | 65 +
src/cfgparse-global.c | 21 ++-
src/cfgparse.c | 3 +-
5 files changed, 98 insertions(+), 2 deletions(-)
create mode 100644 reg-tests/tcp-rules/default_rules_mptcp.vtc
diff --git a/doc/configuration.txt b/doc/configuration.txt
index 16094c194..cd43cd863 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -1341,6 +1341,7 @@ The following keywords are supported in the "global"
section :
- maxsslconn
- maxsslrate
- maxzlibmem
+ - mptcp
- no-memory-trimming
- noepoll
- noevports
@@ -2958,6 +2959,14 @@ maxzlibmem
with "show info" on the line "MaxZlibMemUsage", the memory used by zlib is
"ZlibMemUsage" in bytes.
+mptcp
+ Uses MPTCP instead of TCP. Multipath TCP or MPTCP is an extension to the
+ standard TCP and is described in RFC 8684. It allows a device to make use of
+ multiple interfaces at once to send and receive TCP packets over a single
+ MPTCP connection. MPTCP can aggregate the bandwidth of multiple interfaces or
+ prefer the one with the lowest latency, it also allows a fail-over if one
path
+ is down, and the traffic is seamlessly reinjected on other paths.
+
no-memory-trimming
Disables memory trimming ("malloc_trim") at a few moments where attempts are
made to reclaim lots of memory (on memory shortage or on reload). Trimming
diff --git a/include/haproxy/proto_rhttp.h b/include/haproxy/proto_rhttp.h
index 421680fe5..80d2d6448 100644
--- a/include/haproxy/proto_rhttp.h
+++ b/include/haproxy/proto_rhttp.h
@@ -5,6 +5,8 @@
#include
#include
+extern struct protocol proto_rhttp;
+
int rhttp_bind_receiver(struct receiver *rx, char **errmsg);
int rhttp_bind_listener(struct listener *listener, char *errmsg, int errlen);
diff --git a/reg-tests/tcp-rules/default_rules_mptcp.vtc
b/reg-tests/tcp-rules/default_rules_mptcp.vtc
new file mode 100644
index 0..485d56a73
--- /dev/null
+++ b/reg-tests/tcp-rules/default_rules_mptcp.vtc
@@ -0,0 +1,65 @@
+varnishtest "Test declaration of TCP rules in default sections with mptcp in
the config"
+
+feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
+feature cmd "test \"$(cat /proc/sys/net/mptcp/enabled)\" = 1"
+feature ignore_unknown_macro
+
+server s1 {
+rxreq
+txresp
+expect req.http.x-test1-frt == "def_front"
+expect req.http.x-test1-bck == "def_back"
+} -start
+
+haproxy h1 -conf {
+ global
+ mptcp
+
+ defaults common
+ mode http
+ timeout connect "${HAPROXY_TEST_TIMEOUT-5s}"
+ timeout client
