Status update for 2.1 & 2.0
Hi all, I spent the day working on a few bugs that will need some backporting so I didn't emit 2.1 nor any new dev version since IMHO that's not needed. We still have a few fixes pending for 2.0 that'll make a new 2.0 worth being released, and in addition Christopher backported the ability to alter the outgoing H1 headers' case in HTX mode as rightfully requested in person by a few people at the HaproxyConf to ease their transition to HTX when dealing with bogus applications. So I expect to issue 2.0.10 likely tomorrow as time permits or in the worst case on Monday, and once done we should be all set to release 2.1 on Monday as well. 2.1-dev5 has been working flawlessly on haproxy.org so it's about time to open the cage and let it fly. Have a nice week-end, Willy
Re: Doc for applet/appctx
Hi Aleks, On Fri, Nov 22, 2019 at 07:19:29PM +, Aleksandar Lazic wrote: > > Hi. > > In the code is applet and appctx Quote often used but I haven't seen and > documentation about this part of HAProxy. Is there anything about it and I > haven't seen it? Unfortunately no, that's one of the important things I'd like to document so that occasional or first time contributors are more at ease. In the mean time I can suggest you to look at the CLI's I/O handler if you want to see an incoming applet, or at peers/spoe for outgoing ones. Hoping this helps, Willy
Doc for applet/appctx
Hi. In the code is applet and appctx Quote often used but I haven't seen and documentation about this part of HAProxy. Is there anything about it and I haven't seen it? Regards Alex
Re: [PATCH] MINOR: ssl: deduplicate ca-file
Hi Manu,
I have a few questions/remarks below:
> Subject: [PATCH 1/3] MINOR: ssl: deduplicate ca-file
> [...]
>
> +static int ssl_store_load_locations_file(X509_STORE **store_ptr, char *path)
> +{
> + struct ebmb_node *eb;
> + struct cafile_entry *ca_e;
> +
> + eb = ebst_lookup(_tree, path);
> + if (eb)
> + ca_e = ebmb_entry(eb, struct cafile_entry, node);
> + else {
> + X509_STORE *store = X509_STORE_new();
> + if (X509_STORE_load_locations(store, path, NULL)) {
> + int pathlen;
> + pathlen = strlen(path);
> + ca_e = calloc(1, sizeof(*ca_e) + pathlen + 1);
You probably want to check the return of calloc here.
> + memcpy(ca_e->path, path, pathlen + 1);
> + ca_e->ca_store = store;
> + ebst_insert(_tree, _e->node);
> + } else {
> + X509_STORE_free(store);
> + return 0;
> + }
> + }
> + *store_ptr = ca_e->ca_store;
> + return 1;
> +}
> +
> Subject: [PATCH 2/3] MINOR: ssl: compute ca-list from deduplicate ca-file
> [...]
>
> +/* Duplicate ca_name is tracking with ebtree. It's simplify openssl
> compatibility */
> +static STACK_OF(X509_NAME)* ssl_load_client_ca_file(char *path)
> +{
> [...]
> +
> + skn = sk_X509_NAME_new_null();
> + /* take x509 from cafile_tree */
> + objs = X509_STORE_get0_objects(ca_e->ca_store);
> + for (i = 0; i < sk_X509_OBJECT_num(objs); i++) {
> + x = X509_OBJECT_get0_X509(sk_X509_OBJECT_value(objs,
> i));
> + if (!x)
> + continue;
> + xn = X509_get_subject_name(x);
> + if (!xn)
> + continue;
> + /* Check for duplicates. */
> + key = X509_NAME_hash(xn);
> + for (node = eb64_lookup(_name_tree, key); node; node
> = eb64_next(node)) {
> + ca_name = container_of(node, typeof(*ca_name),
> node);
> + if (X509_NAME_cmp(xn, ca_name->xname) == 0)
> + continue;
> + }
I'm not sure this part is right. I think you wanted to skip pushing the object
if it was already in the tree, but instead you are doing a continue in the
inner loop.
Also I'm not sure why we have to deduplicate the entries? Isn't it the job of
openssl to do that?
> + xn = X509_NAME_dup(xn);
> + sk_X509_NAME_push(skn, xn);
> + ca_name = calloc(1, sizeof *ca_name);
Same issue with calloc there.
> + ca_name->node.key = key;
> + ca_name->xname = xn;
> + eb64_insert(_name_tree, _name->node);
After digging into this part of the code, I think it's safer to split the logic
in two parts, like what has been done for the ckch.
The access to the file system should be done in the configuration parsing, so
we don't access at all to the files in ssl_sock_prepare_ctx().
And we should only have a lookup in ssl_sock_prepare_ctx().
Thanks!
--
William Lallemand
Re: [PATCH] MINOR: contrib/prometheus-exporter: allow to select the exported metrics
On Fri, Nov 22, 2019 at 11:25:44AM +0100, Christopher Faulet wrote: > URI delimiters must not be encoded, except if you want to escape it. So, > your first url is not equivalent to the second one. The encoded question > mark is part of the path in the first url, it is not the query-string > delimiter. wow I learned something today :) and there is indeed a specific parameter for this precise reason you mentioned. The end config looks like: - job_name: 'lb-metrics' metrics_path: '/metrics' params: scope: ['global', 'frontend', 'backend'] consul_sd_configs: - server: localhost:8500 services: - lb relabel_configs: - source_labels: ['__meta_consul_dc'] target_label: 'dc' - source_labels: ['__address__', '__meta_consul_service_id'] target_label: 'instance' and this time it works; I'm reassured :) > But, cause of your remark, I re-read the function code and it revealed a > bug. The query-string is decoded before parsing. Only parameters names and > values must be decoded. Thus, a fix is still required. good, I will handle the fix. Thanks, -- William
Re: HTX no connection close - 2.0.9
Le 21/11/2019 à 23:54, Valters Jansons a écrit : Hello everyone, I am running HAProxy v2.0.9 on Ubuntu using the dedicated PPA (ppa:vbernat/haproxy-2.0). There seems to be a behavior change for a specific endpoint between HTX enabled and HTX disabled, but I have not been able to pin-point the exact root cause. With HTX disabled (`no option http-use-htx`), a browser makes a POST request (ALPN H2) which is shown as HTTP/1.1. That then reaches the backend (IIS) as HTTP/1.1 and finishes successfully in around 10 seconds. With the default behavior of HTX enabled, the POST request comes in and is shown as HTTP/2.0. It then connects to backend as HTTP/1.1 and the client receives a 200 OK and the response data around the same time as without HTX. However, the connection does not get properly closed until server timeout with a termination_state of sD-- (server-side timeout in the DATA phase). At that point, debug log shows `srvcls` and the client connection is 'successfully' closed. The backend itself seems to think it handled the request 'as usual'. The non-HTX debug log does not show srvcls, clicls and closed events on the backend whatsoever, but seeing as that connection does terminate, I am guessing the relevant events just don't get logged with HTX disabled. We are using http-keep-alive as the default connection mode, but changing it to http-server-close or httpclose does not seem to make a difference. The strange part here is that we are seeing this particular behavior with HTX enabled only on browsers (tested Chrome and Firefox on multiple machines), as testing using cURL (H2) or simply via OpenSSL's s_client (HTTP/1.1) appears to work even when HTX is enabled, and additionally, we are seeing this on the particular endpoint only for a specific user's context. That could also imply that it has something to do with the response data, or maybe it could just be a red herring. Maybe HTX is waiting on some trailing headers or some other feature of HTTP.. Any ideas as to where I should start troubleshooting HTX behavior for one production endpoint for one specific user context? Hi, Could you share your configuration please ? If it only happens on a specific endpoint, you can remove configuration of the others. Then if it is easily reproducible, you may try to find the minimal config to do so. Finally a network capture on a server side may help too (share it privately). In the mean time, could you describe your request and your response when the problem occurs (size, chunked-encoding Vs content-length, compression ...) ? And from the browser point of view, is there any difference with and without the HTX ? If possible, could you try disabling the h2 on the frontend side ? It could help to identify where the problem is. Finally, have you already tested other 2.X versions without encountering the problem? Thanks, -- Christopher Faulet
Re: [PATCH] MINOR: ssl: deduplicate ca-file
Fix bad merge from my branch, > Le 22 nov. 2019 à 11:35, Emmanuel Hocdet a écrit : > > > Patches update with compat lib-ssl and crl-file. > Deduplicate Verify-stuff in memory will prevent file access when updating a > certificate with CLI. 0001-MINOR-ssl-deduplicate-ca-file.patch Description: Binary data 0002-MINOR-ssl-compute-ca-list-from-deduplicate-ca-file.patch Description: Binary data 0003-MINOR-ssl-deduplicate-crl-file.patch Description: Binary data
Is it a bug?
Dear haproxy:
I used haproxy as a http proxy, and make data traffic throw it . I send a
TTOU(pause) signal to the woker process of haproxy, then all of the wokers
hanging at the listen_accept (accept) function.
gaohd@nxg_88:~/haproxy$ sudo ./haproxy -f ../gaohd.cfg -W
gaohd@nxg_88:~/script$ ps -auxf |grep haproxy
root 20067 0.0 0.0 62224 4360 pts/1S+ 18:53 0:00 |
\_ sudo ./haproxy -f ../gaohd.cfg -W
root 20068 0.0 0.0 28960 11004 pts/1S+ 18:53 0:00 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20069 0.0 0.0 29212 1020 ?Ss 18:53 0:00 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20070 0.0 0.0 29212 1020 ?Ss 18:53 0:00 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20071 0.0 0.0 29212 1020 ?Ss 18:53 0:00 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20072 0.0 0.0 29212 2440 ?Ss 18:53 0:00 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20073 0.0 0.0 29212 2440 ?Ss 18:53 0:00 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20074 0.0 0.0 29212 2440 ?Ss 18:53 0:00 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20075 0.0 0.0 29212 2440 ?Ss 18:53 0:00 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20076 0.0 0.0 29212 2440 ?Ss 18:53 0:00 |
\_ ./haproxy -f ../gaohd.cfg -W
gaohd@nxg-lab-66:~$ ab -n 200 -c 100 http://www.axddos.com:80/index.html
gaohd@nxg_88:~/script$ sudo kill -s TTOU 20076
gaohd@nxg_88:~/script$ ps -auxf |grep haproxy
root 20067 0.0 0.0 62224 4360 pts/1S+ 18:53 0:00 |
\_ sudo ./haproxy -f ../gaohd.cfg -W
root 20068 0.0 0.0 28960 11004 pts/1S+ 18:53 0:00 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20069 7.7 0.0 29348 3620 ?Ds 18:53 0:02 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20070 7.7 0.0 29348 2504 ?Rs 18:53 0:02 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20071 7.6 0.0 29348 3620 ?Ds 18:53 0:02 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20072 7.7 0.0 29348 2440 ?Ds 18:53 0:02 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20073 7.7 0.0 29352 2440 ?Ds 18:53 0:02 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20074 7.7 0.0 29348 2440 ?Ds 18:53 0:02 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20075 7.7 0.0 29348 3632 ?Rs 18:53 0:02 |
\_ ./haproxy -f ../gaohd.cfg -W
root 20076 7.7 0.0 29352 2440 ?Ds 18:53 0:02 |
\_ ./haproxy -f ../gaohd.cfg -W
I download all the versions of haproxy and found that versions between 1.8.20
to 2.0.9 have this promble.
I resarch the code and found that this problem was introduced at commite
c98cdf7cc755c579a8b9cceee4809089267615ce
--- a/src/listener.c
+++ b/src/listener.c
@@ -644,7+684,14 @@ void listener_accept(int fd)
if (next_conn)
HA_ATOMIC_SUB(>nbconn, 1);
- if (l->nbconn < l->maxconn && l->state == LI_FULL) {
+ if (p && next_feconn)
+ HA_ATOMIC_SUB(>feconn, 1);
+
+ if (next_actconn)
+ HA_ATOMIC_SUB(, 1);
+
+ if ((l->state == LI_FULL && l->nbconn < l->maxconn) ||
+ (l->state == LI_LIMITED && ((!p || p->feconn < p->maxconn) &&
(actconn < global.maxconn { //(The problem is case by this line)
/* at least one thread has to this when quitting */
resume_listener(l);
Is it a bug ? What is the root reason for this bug ? I didn't find out,
could you help me to answer it?
Re: [PATCH] MINOR: ssl: deduplicate ca-file
Hi, > Le 29 oct. 2019 à 07:59, Willy Tarreau a écrit : > > Please, let's revisit this after the release. The only people able to > have a look at this and to have an opinion on it are all busy finishing > this release. > Patches update with compat lib-ssl and crl-file. Deduplicate Verify-stuff in memory will prevent file access when updating a certificate with CLI. ++ Manu 0001-MINOR-ssl-deduplicate-ca-file.patch Description: Binary data 0002-MINOR-ssl-compute-ca-list-from-deduplicate-ca-file.patch Description: Binary data 0003-MINOR-ssl-deduplicate-crl-file.patch Description: Binary data
Re: [PATCH] MINOR: contrib/prometheus-exporter: allow to select the exported metrics
Le 21/11/2019 à 16:54, William Dauchy a écrit :
Hi Christopher,
On Tue, Nov 19, 2019 at 04:35:47PM +0100, Christopher Faulet wrote:
+/* Parse the query stirng of request URI to filter the metrics. It returns 1 on
+ * success and -1 on error. */
+static int promex_parse_uri(struct appctx *appctx, struct stream_interface *si)
+{
+ struct channel *req = si_oc(si);
+ struct channel *res = si_ic(si);
+ struct htx *req_htx, *res_htx;
+ struct htx_sl *sl;
+ const char *p, *end;
+ struct buffer *err;
+ int default_scopes = PROMEX_FL_SCOPE_ALL;
+ int len;
+
+ /* Get the query-string */
+ req_htx = htxbuf(>buf);
+ sl = http_get_stline(req_htx);
+ if (!sl)
+ goto error;
+ p = http_find_param_list(HTX_SL_REQ_UPTR(sl), HTX_SL_REQ_ULEN(sl), '?');
+ if (!p)
+ goto end;
It's my turn to be sorry. I wrongly tested on my side regarding a real
integration with prometheus, because I mixed the old metrics and the new
ones. Indeed, prometheus is trying to scrape the encoded url such as:
metrics%3Fscope=global=frontend=backend
instead of
metrics?scope=global=frontend=backend
Hi William,
For the record, I report here our private conversation.
URI delimiters must not be encoded, except if you want to escape it. So, your
first url is not equivalent to the second one. The encoded question mark is part
of the path in the first url, it is not the query-string delimiter.
But, cause of your remark, I re-read the function code and it revealed a bug.
The query-string is decoded before parsing. Only parameters names and values
must be decoded. Thus, a fix is still required.
--
Christopher Faulet
