Re: Upgrade from 1.7 to 2.0 = increased CPU usage
Hi Lukas, I will send the strace to you and Willy so I don't spam the entire maillist with a 10MB tarball. /Elias On Fri, Jul 19, 2019 at 12:11 AM Lukas Tribus wrote: > Hello Elias, > > > On Wed, 17 Jul 2019 at 17:52, Elias Abacioglu > wrote: > > > > Ok, I just tried HAProxy 2.0.2. > > I see weird CPU behaviour there too. > > On threaded mode: > > When I restart or reload haproxy it goes between 11-38% idle on core 2 > and 3. > > But then after a couple of minutes it goes to using almost 100% of core > 2 and 3. > > Ok, thanks. > > Could you attach strace -tt -p to a process occupying 100% and > provide a few seconds of it’s output (it will be large)? > > There is also a similar report on discourse: > > https://discourse.haproxy.org/t/2-0-1-cpu-usage-at-near-100-after-upgrade-from-1-5/4065/ > > I'm CC'ing Willy. > > > Regards, > > Lukas >
[ANNOUNCE] haproxy-1.9.9
Hi, HAProxy 1.9.9 was released on 2019/07/23. It added 110 new commits after version 1.9.8. This release is large because over the last month several bugs affected 2.0 and likely 1.9 and it was uncertain whether some of the fixes to be backported to 1.9 were partially responsible for those bugs nor were totally correct until 2.0.2 was released. Now that 2.0.2 was released, these doubts have been cleared. I'm not going to rehash all the fixes that were enumerated in 2.0.2 last week, many of which are also here but will only mention the most important ones. First we have a critical issue affecting HTX mode with cookies (stickiness or capture). It is possible to make haproxy enter an infinite loop by sending it an improperly formated cookie header. While in 2.0 this results in the watchdog killing the process, in 1.9 there is no such watchdog so the service becomes unresponsive. The simplest workaround is to not enable HTX mode (it is disabled by default, unless you explicitly enabled it with "option http-use-htx"). Those who enable it generally do so to support gRPC using end-to-end HTTP/2. In this case cookies are normally not used so it shouldn't be a problem to have to disable HTX when cookies are in use and conversely. If for any reason you cannot upgrade nor disable HTX nor disable cookies, there is another (ugly) workaround involving TCP request rules which seems to work but that I'm not posting as I'd rather not encourage people to deploy it. Just raise your hand if you really need it. Another major problem concerns the thread safety when dealing with limited listeners : deadlocks and crashes can happen when the frontend's or process's maxconn were reached on multiple threads and a connection is released by another thread. One workaround may simply consist in disabling threads or significantly raising the frontend's maxconn value. Another bug was affecting data forwarding in HTTP/1 in HTX mode. A mistake in the trash management when trying to limit the amount of copies could result in corrupted responses to be returned depending on the usage ratio of the buffers. The last major bug may trigger with threads when a server fails to accept a connection, then a redispatch happens and in the mean time the selected server becomes full and finally cannot accept the connection anymore. In this case we can enter an infinite loop trying to avoid this server. Most of the other bugs concern locking issues (when threads are enabled), connection issues (CLOSE_WAIT mainly and unhandled events causing 100% CPU while still serving the traffic), and risks of truncated payloads in HTX mode, 1xx response handling in HTX, and various issues affecting health checks that were already covered in great lengths in 2.0.x announces. Given the diversity of all the bugs fixed in this version, all users of 1.9 should upgrade, even if not affected by the security issue. Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/1.9/src/ Git repository : http://git.haproxy.org/git/haproxy-1.9.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-1.9.git Changelog: http://www.haproxy.org/download/1.9/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ Willy --- Complete changelog : Ben51Degrees (2): MEDIUM: 51d: Enabled multi threaded operation in the 51Degrees module. BUG/MINOR: 51d/htx: The _51d_fetch method, and the methods it calls are now HTX aware. Christopher Faulet (47): BUG/MINOR: http_fetch: Rely on the smp direction for "cookie()" and "hdr()" BUG/MEDIUM: spoe: Don't use the SPOE applet after releasing it BUG/MINOR: mux-h2: Count EOM in bytes sent when a HEADERS frame is formatted BUG/MINOR: mux-h1: Report EOI instead EOS on parsing error or H2 upgrade BUG/MEDIUM: proto-htx: Not forward too much data when 1xx reponses are handled BUG/MINOR: htx: Remove a forgotten while loop in htx_defrag() BUG/MEDIUM: mux-h1: Don't switch the mux in BUSY mode on 1xx messages BUG/MINOR: mux-h1: Wake up the mux if it is busy when a 1xx response is handled BUG/MEDIUM: mux-h1: Don't skip the TCP splicing when there is no more data to read BUG/MINOR: channel/htx: Don't alter channel during forward for empty HTX message BUG/MINOR: mux-h1: errflag must be set on H1S and not H1M during output processing BUG/MINOR: mux-h1: Don't send more data than expected BUG/MEDIUM: compression/htx: Fix the adding of the last data block BUG/MINOR: channel/htx: Call channel_htx_full() from channel_full() BUG/MINOR: http: Use the global value to limit the number of parsed headers BUG/MEDIUM: proto_htx: Introduce the state ENDING during forwarding
[ANNOUNCE] haproxy-2.0.3 (Security update)
Hi, HAProxy 2.0.3 was released on 2019/07/23. It added 19 new commits after version 2.0.2. This release contains a security fix in the HTX analyzers, in the code responsible of the client-side or the server-side cookies parsing. The bug happens when a Cookie or a Set-Cookie header value starts by a delimiter (a colon or a semicolon). In such case, HAProxy enters in an infinte loop. Depending on the operating system, the service may become degraded, unresponsive, or may trigger haproxy's watchdog causing a service stop or automatic restart. CVE-2019-14241 was assigned to this bug. It contains a configuration workaround. It could help to quickly mitigate the bug For those who want to quickly deploy a fix, please use the following patch for the 2.0 : http://git.haproxy.org/?p=haproxy-2.0.git;a=commitdiff;h=fc7f52eb0 Outside from this critical bug, some other issues were fixed here and there: * In the H1 multiplexer, if some pending data remains on the server-side at the end of the transaction, the server connection is now systematically closed to be sure to not use these data as the response for the next transaction. Most of time, the reponse is be invalid and the client receives a 502 error message. * Two medium bugs on tcp-check were fixed. The first one happens when the function tcpcheck_main() is called while the connection is already subscribed for receiving. In such situation, there isq no reason to try to receive more data again. The second one leads to a crash of HAProxy because of a NULL pointer dereference when tcpcheck_main() is called with no conn_stream and a known check result. * The option http_proxy is fixed. The bug was introduced during 1.8-rc1 development. The temporary connection used to hold the target connection's address was missing a valid target, resulting in a 500 server error being reported when trying to connect to a remote host. * Several minor bugs on the HTX were fixed. It is now possible to have empty error files to bypass default messages without sending anything, just like in the legacy HTTP mode. The sample fetches http_auth() and http_auth_group() are fixed to use a valid HTX message when called from TCP rules for a TCP proxy. The lua function txn:done() is now HTX aware. And finally, the cache filter is now able to parse the header "Cache-Control" on the response to find "max-age" or "s-maxage" values. * When a connection is accepted, if the session accept fails, we take care to have an H1 connection to return a 500 error message to the client. We also return a default HTTP error message at this stage to be sure to not use a chunk owning an HTX message instead of a raw HTTP message. So for anyone using HAproxy-2.0 or considering trying it, please use the release 2.0.3. Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/2.0/src/ Git repository : http://git.haproxy.org/git/haproxy-2.0.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.0.git Changelog: http://www.haproxy.org/download/2.0/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Christopher Faulet (12): BUG/MINOR: http_fetch: Fix http_auth/http_auth_group when called from TCP rules BUG/MINOR: http_htx: Initialize HTX error messages for TCP proxies BUG/MINOR: cache/htx: Make maxage calculation HTX aware BUG/MINOR: hlua: Make the function txn:done() HTX aware DOC: htx: Update comments in HTX files BUG/MINOR: debug: Remove flags CO_FL_SOCK_WR_ENA/CO_FL_SOCK_RD_ENA BUG/MINOR: session: Emit an HTTP error if accept fails only for H1 connection BUG/MINOR: session: Send a default HTTP error if accept fails for a H1 socket BUG/MEDIUM: mux-h1: Trim excess server data at the end of a transaction BUG/MINOR: mux-h1: Close server connection if input data remains in h1_detach() BUG/MINOR: http_ana: Be sure to have an allocated buffer to generate an error BUG/MINOR: http_htx: Support empty errorfiles Olivier Houchard (2): BUG/MEDIUM: checks: Don't attempt to receive data if we already subscribed. BUG/CRITICAL: http_ana: Fix parsing of malformed cookies which start by a delimiter Willy Tarreau (5): BUG/MINOR: dns: remove irrelevant dependency on a client connection BUG/MEDIUM: http/htx: unbreak option http_proxy BUG/MINOR: backend: do not try to install a mux when the connection failed BUG/MINOR: checks: do not exit tcp-checks from the middle of the loop BUG/MEDIUM: tcp-checks: do not dereference inexisting conn_stream -- Christopher Faulet
Re: [PATCH] MINOR: contrib/spoa_server: Port SPOA server to Python 3
Hey,
On 23. 07. 2019. 15:47, Aleksandar Lazic wrote:
> Hi.
>
> Am 23.07.2019 um 15:34 schrieb Vedran Furac:
>> Hello,
>>
>> Attached is a simple port of SPOA with embedded Python 2 interpreter to
>> Python 3
>> following the official Python porting documentation. I tried to keep the
>> changes
>> at minimum. Tested on my local Debian with Python v3.7 using the provided
>> example script.
>
> Please can you also adopt the README to reflect this change.
Thanks, updated the README, amended the commit and attached new patch.
Thanks,
Vedran
>From 360387fce1c97d36d7a6c9a7cb38c82f4ab51592 Mon Sep 17 00:00:00 2001
From: Vedran Furac
Date: Fri, 19 Jul 2019 15:18:56 +0200
Subject: [PATCH] MINOR: contrib/spoa_server: Port SPOA server to Python 3
Python 2 will be EOL in 2020. This patch ports SPOA server with
embedded interpreter to Python 3
---
contrib/spoa_server/Makefile| 6 +-
contrib/spoa_server/README | 2 +-
contrib/spoa_server/ps_python.c | 123 ++--
3 files changed, 74 insertions(+), 57 deletions(-)
diff --git a/contrib/spoa_server/Makefile b/contrib/spoa_server/Makefile
index f0752829..70ccf988 100644
--- a/contrib/spoa_server/Makefile
+++ b/contrib/spoa_server/Makefile
@@ -23,12 +23,12 @@ endif
ifneq ($(USE_PYTHON),)
OBJS += ps_python.o
-CFLAGS += -I/usr/include/python2.7
-LDLIBS += -lpython2.7
+CFLAGS += $(shell python3-config --cflags) -Wno-sign-compare -fPIC
+LDFLAGS += $(shell python3-config --ldflags)
endif
spoa: $(OBJS)
- $(LD) $(LDFLAGS) -o $@ $^ $(LDLIBS)
+ $(LD) $(LDFLAGS) -o $@ $^
install: spoa
install spoa $(DESTDIR)$(BINDIR)
diff --git a/contrib/spoa_server/README b/contrib/spoa_server/README
index fa039c1a..f8ada376 100644
--- a/contrib/spoa_server/README
+++ b/contrib/spoa_server/README
@@ -16,7 +16,7 @@ distribution repositories or from the source.
CentOS/RHEL: yum install python-devel
-The current python version in use is 2.7.
+The current python version in use is 3.x.
Compilation
diff --git a/contrib/spoa_server/ps_python.c b/contrib/spoa_server/ps_python.c
index 0a9fbffc..654bebdf 100644
--- a/contrib/spoa_server/ps_python.c
+++ b/contrib/spoa_server/ps_python.c
@@ -36,6 +36,10 @@ static int ps_python_start_worker(struct worker *w);
static int ps_python_load_file(struct worker *w, const char *file);
static int ps_python_exec_message(struct worker *w, void *ref, int nargs, struct spoe_kv *args);
+struct module_state {
+PyObject *error;
+};
+
static struct ps ps_python_bindings = {
.init_worker = ps_python_start_worker,
.load_file = ps_python_load_file,
@@ -171,11 +175,11 @@ static PyObject *ps_python_set_var_ipv4(PyObject *self, PyObject *args)
value = PyObject_GetAttrString(ipv4, "packed");
if (value == NULL)
return NULL;
- if (PyString_GET_SIZE(value) != sizeof(ip)) {
- PyErr_Format(spoa_error, "UPv6 manipulation internal error");
+ if (PyBytes_Size(value) != sizeof(ip)) {
+ PyErr_Format(spoa_error, "IPv4 manipulation internal error");
return NULL;
}
- memcpy(, PyString_AS_STRING(value), PyString_GET_SIZE(value));
+ memcpy(, PyBytes_AsString(value), PyBytes_Size(value));
if (!set_var_ipv4(worker, name, name_len, scope, )) {
PyErr_SetString(spoa_error, "No space left available");
return NULL;
@@ -202,11 +206,11 @@ static PyObject *ps_python_set_var_ipv6(PyObject *self, PyObject *args)
value = PyObject_GetAttrString(ipv6, "packed");
if (value == NULL)
return NULL;
- if (PyString_GET_SIZE(value) != sizeof(ip)) {
- PyErr_Format(spoa_error, "UPv6 manipulation internal error");
+ if (PyBytes_Size(value) != sizeof(ip)) {
+ PyErr_Format(spoa_error, "IPv6 manipulation internal error");
return NULL;
}
- memcpy(, PyString_AS_STRING(value), PyString_GET_SIZE(value));
+ memcpy(, PyBytes_AsString(value), PyBytes_Size(value));
if (!set_var_ipv6(worker, name, name_len, scope, )) {
PyErr_SetString(spoa_error, "No space left available");
return NULL;
@@ -272,55 +276,31 @@ static PyMethodDef spoa_methods[] = {
"Set SPOA str variable"},
{"set_var_bin", ps_python_set_var_bin, METH_VARARGS,
"Set SPOA bin variable"},
- { /* end */ }
+ {NULL, NULL}
};
-static int ps_python_start_worker(struct worker *w)
+static struct PyModuleDef spoa_module = {
+PyModuleDef_HEAD_INIT,
+"spoa",
+NULL,
+sizeof(struct module_state),
+spoa_methods,
+NULL,
+NULL,
+NULL,
+NULL
+};
+
+static PyObject *PyInit_spoa(void)
{
- PyObject *m;
- PyObject *module_name;
- PyObject *value;
int ret;
-
- Py_SetProgramName("spoa-server");
- Py_Initialize();
-
- module_name = PyString_FromString("ipaddress");
- if (module_name == NULL) {
- PyErr_Print();
- return 0;
- }
-
- module_ipaddress = PyImport_Import(module_name);
- Py_DECREF(module_name);
- if (module_ipaddress == NULL) {
- PyErr_Print();
- return 0;
- }
-
- ipv4_address = PyObject_GetAttrString(module_ipaddress, "IPv4Address");
- if (ipv4_address == NULL) {
-
Re: Upgrade from 1.7 to 2.0 = increased CPU usage
Hello Elias, could you try 2.0.3 please? It was just released today and fixes a CPU hogging issue. cheers, lukas
Re: FreeBSD CI builds fail
Hi guys, On Tue, Jul 23, 2019 at 08:37:37PM +0200, Jerome Magnin wrote: > On Tue, Jul 23, 2019 at 07:09:57PM +0200, Tim Düsterhus wrote: > > Jérôme, > > Ilya, > > > > I noticed that FreeBSD CI fails since > > https://github.com/haproxy/haproxy/commit/885f64fb6da0a349dd3182d21d337b528225c517. > > > > > > One example is here: https://github.com/haproxy/haproxy/runs/169980019 I also noticed the build failure but couldn't find any link to the build history to figure when it started to fail. How did you figure that the commit above was the first one ? > This one fails because there's a L4 timeout, I can probably update the regex > to > take that into account, the interesting part is the failure and the step at > which it fails, but for now we expect a connection failure and not a timeout. There's always the possibility (especially in CI environments) that some rules are in place on the system to prevent connections to unexpected ports. Ideally we'd need a level of failure in CI builds. Some should be just of level "info" and not cause a build error because we'd know they are likely to fail but are still interested in the logs. But I don't think we can do this. Willy
reg-tests/checks/tcp-check_multiple_ports.vtc fails on FreeBSD
can someone have a look ? https://cirrus-ci.com/task/5737816791711744
FreeBSD CI builds fail
Jérôme, Ilya, I noticed that FreeBSD CI fails since https://github.com/haproxy/haproxy/commit/885f64fb6da0a349dd3182d21d337b528225c517. One example is here: https://github.com/haproxy/haproxy/runs/169980019 It should be investigated whether the reg-test is valid for FreeBSD and either be fixed or disabled. Best regards Tim Düsterhus
Re: FreeBSD CI builds fail
On Tue, Jul 23, 2019 at 07:09:57PM +0200, Tim Düsterhus wrote: > Jérôme, > Ilya, > > I noticed that FreeBSD CI fails since > https://github.com/haproxy/haproxy/commit/885f64fb6da0a349dd3182d21d337b528225c517. > > > One example is here: https://github.com/haproxy/haproxy/runs/169980019 > > It should be investigated whether the reg-test is valid for FreeBSD and > either be fixed or disabled. > > Best regards > Tim Düsterhus > Thanks Tim and Ilya, This one fails because there's a L4 timeout, I can probably update the regex to take that into account, the interesting part is the failure and the step at which it fails, but for now we expect a connection failure and not a timeout. I'm a bit more concerned about the other one reported by Ilya where the backend server started by VTest won't accept connection. I'll look into this one further. Jérôme
