timeout queue broken since f6e6dc1
Just ran across an issue where the `timeout queue` option is non-functional. I can send a request to haproxy which sits in the queue for well past the configured limit. It appears the issue popped up as a result of this commit: commit f6e6dc12cd533b2d8bb6413a4b5f875ddfd3e6e3 (refs/bisect/bad) Author: Olivier Houchard Date: Fri May 18 18:38:23 2018 +0200 MAJOR: tasks: Create a per-thread runqueue. A lot of tasks are run on one thread only, so instead of having them all in the global runqueue, create a per-thread runqueue which doesn't require any locking, and add all tasks belonging to only one thread to the corresponding runqueue. The global runqueue is still used for non-local tasks, and is visited by each thread when checking its own runqueue. The nice parameter is thus used both in the global runqueue and in the local ones. The rare tasks that are bound to multiple threads will have their nice value used twice (once for the global queue, once for the thread-local one). Reproduced with TARGET=osx Compiler clang-900.0.39.2 -Patrick
haproxy requests hanging since b0bdae7
It seems that commit b0bdae7 has completely broken haproxy for me. When I send a request to haproxy, it just sits there. The backend server receives nothing, and the client waits for a response. Running with debug enabled I see just a single line: :f1.accept(0004)=0005 from [127.0.0.1:63663] ALPN= commit b0bdae7b88d53cf8f18af0deab6d4c29ac25b7f9 (refs/bisect/bad) Author: Olivier Houchard Date: Fri May 18 18:45:28 2018 +0200 MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible). Issue reproducible with a very simple config: defaults mode http frontend f1 bind :8081 default_backend b1 backend b1 server s1 127.0.0.1:8081 Compiled on OS-X with only a single make variable of TARGET=osx Compiler: clang-900.0.39.2 -Patrick
Re: HAProxy - Server Timeout and Client Timeout
HI Michael We often see the client/server timeouts requiring to be raised as you have found, A good default value for client/server timeouts are below and I include the connect timeout too in milliseconds timeout connect 4000 timeout client 42000 timeout server 43000 Say for example you run a long report on a website, maybe it will take 5 or 10 min's to complete so you would raise the server timeout to match the required maximum timeout where possible allowing for the report to run and the server, not timeout. I hope this helps Andruw Smalley Loadbalancer.org Ltd. www.loadbalancer.org +1 888 867 9504 / +44 (0)330 380 1064 asmal...@loadbalancer.org Leave a Review | Deployment Guides | Blog On 5 June 2018 at 20:11, Martel, Michael H. wrote: > Greetings! > > We're running HAproxy 1.5.18 on RedHat Enterprise 7.4, as the load balancer > for our LMS (Moodle). We have found that the course backup feature in Moodle > will return a 5xx error on some backups. We have determined that the > "timeout server" value needed to be increased. > > Initially we were using a "timeout client 1m" and "timeout server 1m" . > Adjusting the server to "timeout server 12m" fixes the problem and does not > appear to introduce any other issues in our testing. > > I can't see any reason that I should have the "timeout client" and the > "timeout server" set to the same value. > > Is there anything I should watch out for after increasing the "timeout > server" by such a large amount ? > > Thanks! > > > > Michael > > -- > > o- >Michael H. Martel | Director of Data Center Administration >michael.mar...@vsc.edu | Systems and Security Administrator >Vermont State Colleges | PH:802-224-3010 FX:802-224-3035 > >
HAProxy - Server Timeout and Client Timeout
Greetings! We're running HAproxy 1.5.18 on RedHat Enterprise 7.4, as the load balancer for our LMS (Moodle). We have found that the course backup feature in Moodle will return a 5xx error on some backups. We have determined that the "timeout server" value needed to be increased. Initially we were using a "timeout client 1m" and "timeout server 1m" . Adjusting the server to "timeout server 12m" fixes the problem and does not appear to introduce any other issues in our testing. I can't see any reason that I should have the "timeout client" and the "timeout server" set to the same value. Is there anything I should watch out for after increasing the "timeout server" by such a large amount ? Thanks! Michael -- o- Michael H. Martel | Director of Data Center Administration michael.mar...@vsc.edu | Systems and Security Administrator Vermont State Colleges | PH:802-224-3010 FX:802-224-3035
stable-bot: NOTICE: 10 bug fixes in queue for next release
Hi, This is a friendly bot that watches fixes pending for the next haproxy-stable release! One such e-mail is sent every week once patches are waiting in the last maintenance branch, and an ideal release date is computed based on the severity of these fixes and their merge date. Responses to this mail must be sent to the mailing list. Last release 1.8.9 was issued on 2018/05/18. There are currently 10 patches in the queue cut down this way: - 2 BUILD, first one merged on 2018/05/23 - 6 MEDIUM, first one merged on 2018/05/23 - 2 MINOR, first one merged on 2018/05/23 Thus the computed ideal release date for 1.8.10 would be 2018/06/20, which is in two weeks or less. The current list of patches in the queue is: - BUILD : fd: fix typo causing a warning when threads are disabled - BUILD : threads: unbreak build without threads - MEDIUM : spoe: Flags are not encoded in network order - MEDIUM : fd: Only check update_mask against all_threads_mask. - MEDIUM : cache: don't cache when an Authorization header is present - MEDIUM : contrib/mod_defender: Use network order to encode/decode flags - MEDIUM : dns: Delay the attempt to run a DNS resolution on check failure. - MEDIUM : contrib/modsecurity: Use network order to encode/decode flags - MINOR : lua: Socket.send threw runtime error: 'close' needs 1 arguments. - MINOR : ssl/lua: prevent lua from affecting automatic maxconn computation --- The haproxy stable-bot is freely provided by HAProxy Technologies to help improve the quality of each HAProxy release. If you have any issue with these emails or if you want to suggest some improvements, please post them on the list so that the solutions suiting the most users can be found.
HAProxy 1.8.x not serving errorfiles with H2
We are in the process of testing HAProxy 1.8.x with ALPN and H2 on some of our servers. We have default 502 and 503 errorfiles defined (ex. errorfile 503 /etc/haproxy/errors/503.http), but we've noticed that these errorfiles are not served to the user's browser when the error occurs (for instance, if the backend is down, a user should get the 503 errorfile). Chrome returns "ERR_SPDY_PROTOCOL_ERROR", Curl [1] returns "curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)", and Firefox shows "The connection to was interrupted while the page was loading." With debug logging turned on, I can see that HAProxy is recognizing a 503 if the back-end server is down [2], but it doesn't seem to pass that error through to the client browser. If the backend is up and a 502 is generated, users do not receive the errorfile either. If we turn off H2 and drop back to HTTP/1.1, the errorfiles are displayed properly (though via HTTP/0.9) This has been observed in both 1.8.4 and 1.8.9. Our platform is Amazon Linux, using openssl-1.0.2k-12.109.amzn1.x86_64. Thanks in advance for any thoughts you might have - [1] Curl verbose (curl -I) output: * Trying ... * TCP_NODELAY set * Connected to () port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Client hello (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Client hello (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: [removed] * start date: Mar 20 00:00:00 2017 GMT * expire date: Mar 24 12:00:00 2020 GMT * subjectAltName: host "" matched cert's "" * issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x7fbded005400) > HEAD /libs/cq/core/content/welcome.html HTTP/2 > Host: > User-Agent: curl/7.54.0 > Accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS updated)! * HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2) * Closing connection 0 * TLSv1.2 (OUT), TLS alert, Client hello (1): curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2) [2] haproxy[19803]: :63832 [05/Jun/2018:15:36:24.202] incoming_https~ local_author_app_http/ 0/-1/-1/-1/0 503 1441 - - SCDN 3/1/0/0/0 0/0 "GET /libs/cq/core/content/welcome.html HTTP/1.1"
Re: srv_is_up : unable to find server.
Thanks for replying Im trying to get haproxy to monitor redis-sentinel / redis and see which one redis instance is the master so that traffic is sent there. As originally, per this example https://www.haproxy.com/blog/haproxy-advanced-redis-health-check/ (See the comment January 7, 2018 at 6:52) Regards Brent On 05/06/2018 13:43, Lukas Tribus wrote: On 5 June 2018 at 13:18, Brent Clark wrote: Good day Guys I am at a total loss, and Im hoping someone on this list, would be so kind to review my setup. I am trying to get haproxy to monitor redis / sentinel. But I keep getting. [WARNING] 155/110602 (309) : config : log format ignored for frontend 'ft_redis' since it has no log address. [ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:29] : unable to find server '10.42.131.120' in proxy 'bk_redis', referenced in arg 1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'. [ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:30] : unable to find server '10.42.40.236' in proxy 'bk_redis', referenced in arg 1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'. [ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:31] : unable to find server '10.42.224.133' in proxy 'bk_redis', referenced in arg 1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'. [ALERT] 155/110602 (309) : Fatal errors found in configuration. What I cant understand is, I changed to ips as opposed to hostnames. But haproxy still cant see the peer. Here is my configuration file. https://pastebin.com/raw/DGTsNRDs If someone can assist it would be appreciated. I don't understand what it is you are trying to achieve, none of what you configured makes sense to me. Can you elaborate what you expect haproxy to do and why you need all those backends and use-server directives? Regards, Lukas
Re: [PATCH]: MINOR :task another explicit cast
On Tue, Jun 05, 2018 at 12:49:34PM +0200, Olivier Houchard wrote: > Oops, thanks a lot David, I hope it'll be the last one :) > > Willy, can you please push it ? Now applied, and after checking the code it looks OK now. Thanks guys, Willy
Re: srv_is_up : unable to find server.
Thanks Jerome
I just see this setup goes in line with that you are saying.
https://yemaosheng.com/2016/04/haproxy-cfg-for-redis-sentinel/
Thanks so much for replying.
Regards
Brent
On 05/06/2018 13:49, Jerome Magnin wrote:
Hi Brent,
On Tue, Jun 05, 2018 at 01:18:36PM +0200, Brent Clark wrote:
Good day Guys
I am at a total loss, and Im hoping someone on this list, would be so kind
to review my setup.
I am trying to get haproxy to monitor redis / sentinel. But I keep getting.
[WARNING] 155/110602 (309) : config : log format ignored for frontend
'ft_redis' since it has no log address.
[ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:29] :
unable to find server '10.42.131.120' in proxy 'bk_redis', referenced in arg
1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'.
[ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:30] :
unable to find server '10.42.40.236' in proxy 'bk_redis', referenced in arg
1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'.
[ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:31] :
unable to find server '10.42.224.133' in proxy 'bk_redis', referenced in arg
1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'.
[ALERT] 155/110602 (309) : Fatal errors found in configuration.
What I cant understand is, I changed to ips as opposed to hostnames. But
haproxy still cant see the peer.
Here is my configuration file.
https://pastebin.com/raw/DGTsNRDs
If someone can assist it would be appreciated.
srv_is_up takes an optionnal backend name and a mandatory server name as
argument. server name is the second argument on a server line, it does not have
to be a (resolvable) fqdn.
example:
use-server redis-server-0 if { srv_is_up(10.42.131.120/sentinel0) } ...
I'm not sure I understand what you want to do, though.
Re: srv_is_up : unable to find server.
Hi Brent,
On Tue, Jun 05, 2018 at 01:18:36PM +0200, Brent Clark wrote:
> Good day Guys
>
> I am at a total loss, and Im hoping someone on this list, would be so kind
> to review my setup.
>
> I am trying to get haproxy to monitor redis / sentinel. But I keep getting.
>
> [WARNING] 155/110602 (309) : config : log format ignored for frontend
> 'ft_redis' since it has no log address.
> [ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:29] :
> unable to find server '10.42.131.120' in proxy 'bk_redis', referenced in arg
> 1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'.
> [ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:30] :
> unable to find server '10.42.40.236' in proxy 'bk_redis', referenced in arg
> 1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'.
> [ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:31] :
> unable to find server '10.42.224.133' in proxy 'bk_redis', referenced in arg
> 1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'.
> [ALERT] 155/110602 (309) : Fatal errors found in configuration.
>
> What I cant understand is, I changed to ips as opposed to hostnames. But
> haproxy still cant see the peer.
>
> Here is my configuration file.
> https://pastebin.com/raw/DGTsNRDs
>
> If someone can assist it would be appreciated.
>
srv_is_up takes an optionnal backend name and a mandatory server name as
argument. server name is the second argument on a server line, it does not have
to be a (resolvable) fqdn.
example:
use-server redis-server-0 if { srv_is_up(10.42.131.120/sentinel0) } ...
I'm not sure I understand what you want to do, though.
--
Jérôme
Re: srv_is_up : unable to find server.
On 5 June 2018 at 13:18, Brent Clark wrote: > Good day Guys > > I am at a total loss, and Im hoping someone on this list, would be so kind > to review my setup. > > I am trying to get haproxy to monitor redis / sentinel. But I keep getting. > > [WARNING] 155/110602 (309) : config : log format ignored for frontend > 'ft_redis' since it has no log address. > [ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:29] : > unable to find server '10.42.131.120' in proxy 'bk_redis', referenced in arg > 1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'. > [ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:30] : > unable to find server '10.42.40.236' in proxy 'bk_redis', referenced in arg > 1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'. > [ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:31] : > unable to find server '10.42.224.133' in proxy 'bk_redis', referenced in arg > 1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'. > [ALERT] 155/110602 (309) : Fatal errors found in configuration. > > What I cant understand is, I changed to ips as opposed to hostnames. But > haproxy still cant see the peer. > > Here is my configuration file. > https://pastebin.com/raw/DGTsNRDs > > If someone can assist it would be appreciated. I don't understand what it is you are trying to achieve, none of what you configured makes sense to me. Can you elaborate what you expect haproxy to do and why you need all those backends and use-server directives? Regards, Lukas
srv_is_up : unable to find server.
Good day Guys I am at a total loss, and Im hoping someone on this list, would be so kind to review my setup. I am trying to get haproxy to monitor redis / sentinel. But I keep getting. [WARNING] 155/110602 (309) : config : log format ignored for frontend 'ft_redis' since it has no log address. [ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:29] : unable to find server '10.42.131.120' in proxy 'bk_redis', referenced in arg 1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'. [ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:30] : unable to find server '10.42.40.236' in proxy 'bk_redis', referenced in arg 1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'. [ALERT] 155/110602 (309) : parsing [/usr/local/etc/haproxy/haproxy.cfg:31] : unable to find server '10.42.224.133' in proxy 'bk_redis', referenced in arg 1 of ACL keyword 'srv_is_up' in proxy 'bk_redis'. [ALERT] 155/110602 (309) : Fatal errors found in configuration. What I cant understand is, I changed to ips as opposed to hostnames. But haproxy still cant see the peer. Here is my configuration file. https://pastebin.com/raw/DGTsNRDs If someone can assist it would be appreciated. Kind Regards Brent Clark
Re: Dynamically adding/deleting SSL certificates
On Fri, Jun 1, 2018 at 11:13 AM, Aurélien Nephtali wrote: > > We also need to agree on the payload format to use in the add command: > only the PEM certificate is supported at the moment but when there > will be OCSP + SCTL support it will become messy very quick. > In my tests I am using something like "cert=[...] ocsp=[...] > issuer=[...] sctl=[...]" but it is not pretty. > I thought of using an INI file format but it is not very handy if you > have to craft a file just for one operation. Another idea would be to add a binary protocol to the CLI and distribute a tool that would implement this protocol. The add command would be the first to leverage this protocol to easily upload certificates and all other stuff that may come with it. The CLI parser would switch in binary parsing when receiving a special command (or a special binary pattern). Having two incompatible ways to speak to the software can be confusing but as socat is required to speak to haproxy, using another tool may not be that crazy. -- Aurélien Nephtali
Re: [PATCH]: MINOR :task another explicit cast
Hi, On Tue, Jun 05, 2018 at 10:46:34AM +, David CARLIER wrote: > Hi, > > Did a full rebuild and caught it only. > > Regards. Oops, thanks a lot David, I hope it'll be the last one :) Willy, can you please push it ? Thanks ! Olivier
[PATCH]: MINOR :task another explicit cast
Hi,
Did a full rebuild and caught it only.
Regards.
From c8cda198a8315d780f11ebf03b44a7d2f686f733 Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Tue, 5 Jun 2018 10:41:03 +
Subject: [PATCH] MINOR: task: Fix compiler warning.
Waking up task, when checking if it is a valid entry.
Similarly to commit caa8a37ffe5922efda7fd7b882e96964b40d7135,
casting explicitally to void pointer as HA_ATOMIC_CAS needs.
---
src/task.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/task.c b/src/task.c
index fb484073..16c72323 100644
--- a/src/task.c
+++ b/src/task.c
@@ -79,7 +79,7 @@ void __task_wakeup(struct task *t, struct eb_root *root)
* in the meanwhile.
*/
redo:
- if (unlikely(!HA_ATOMIC_CAS(>rq.node.leaf_p, , 0x1))) {
+ if (unlikely(!HA_ATOMIC_CAS(>rq.node.leaf_p, , (void *)0x1))) {
if (root == )
HA_SPIN_UNLOCK(TASK_RQ_LOCK, _lock);
return;
--
2.17.0
