Hey everyone.
As you may know, our connection timeouts to kojipkgs is back
( https://pagure.io/fedora-infrastructure/issue/12814 )
I have been unable to find a fix yet, but I have a few things I would
like to try:
1. I'd like to try adding:
retries 5
retry-on all-retryable-errors
option redispatch 1
to the kojipkgs backend in haproxy.
This will not fix anything, but it should make it so when a connection
times out it gets sent to the other server and might have a chance of
getting properly served instead of returning a 503. ie, bandaid over the
problem while we try and track it down.
patch:
diff --git a/roles/haproxy/templates/haproxy.cfg
b/roles/haproxy/templates/haproxy.cfg
index c311c0f9d8..f2ba4654e7 100644
--- a/roles/haproxy/templates/haproxy.cfg
+++ b/roles/haproxy/templates/haproxy.cfg
@@ -277,6 +277,9 @@ backend kojipkgs-backend
server kojipkgs01.{{ datacenter }}.fedoraproject.org kojipkgs01.{{
datacenter }}.fedoraproject.org:80 check inter 30s rise 1 fall 3
server kojipkgs02.{{ datacenter }}.fedoraproject.org kojipkgs02.{{
datacenter }}.fedoraproject.org:80 check inter 30s rise 1 fall 3
option httpchk GET /
+ retries 5
+ retry-on all-retryable-errors
+ option redispatch 1
{% endif %}
{% if datacenter == "rdu3" %}
2. I would like to try and take varnish out of the path to see if it's
related to the problem. To do this on kojipkgs01:
- Take kojipkgs01 out of haproxy so it gets no requests
- stop varnish and httpd on it
- reset httpd to listen on port 80 instead of 8080
- confirm it's working
- re-enable in haproxy to get traffic.
If the problem persists we know it's not varnish related.
If it doesn't we know to focus on varnish.
I may have other things to try as I think of them.
kevin
--
_______________________________________________
infrastructure mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[email protected]
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
