On Wed, 2025-10-08 at 10:00 -0700, Kevin Fenzi via infrastructure wrote: > Hey everyone. > > As you may know, our connection timeouts to kojipkgs is back > ( https://pagure.io/fedora-infrastructure/issue/12814 ) > > I have been unable to find a fix yet, but I have a few things I would > like to try: > > 1. I'd like to try adding: > > retries 5 > retry-on all-retryable-errors > option redispatch 1 > > to the kojipkgs backend in haproxy. > This will not fix anything, but it should make it so when a connection > times out it gets sent to the other server and might have a chance of > getting properly served instead of returning a 503. ie, bandaid over the > problem while we try and track it down. > > patch: > > diff --git a/roles/haproxy/templates/haproxy.cfg > b/roles/haproxy/templates/haproxy.cfg > index c311c0f9d8..f2ba4654e7 100644 > --- a/roles/haproxy/templates/haproxy.cfg > +++ b/roles/haproxy/templates/haproxy.cfg > @@ -277,6 +277,9 @@ backend kojipkgs-backend > server kojipkgs01.{{ datacenter }}.fedoraproject.org kojipkgs01.{{ > datacenter }}.fedoraproject.org:80 check inter 30s rise 1 fall 3 > server kojipkgs02.{{ datacenter }}.fedoraproject.org kojipkgs02.{{ > datacenter }}.fedoraproject.org:80 check inter 30s rise 1 fall 3 > option httpchk GET / > + retries 5 > + retry-on all-retryable-errors > + option redispatch 1 > {% endif %} > > {% if datacenter == "rdu3" %} > > 2. I would like to try and take varnish out of the path to see if it's > related to the problem. To do this on kojipkgs01: > > - Take kojipkgs01 out of haproxy so it gets no requests > - stop varnish and httpd on it > - reset httpd to listen on port 80 instead of 8080 > - confirm it's working > - re-enable in haproxy to get traffic. > > If the problem persists we know it's not varnish related. > If it doesn't we know to focus on varnish. > > I may have other things to try as I think of them. > > kevin
+1. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @[email protected] https://www.happyassassin.net -- _______________________________________________ infrastructure mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
