TL;DR: 1984 is having issues with some of their infrastructure, I've removed one of their DNS servers that seems down from winston.parabola.nu's network config.
The process: - Try to install something with pacman - Get timeouts - Discover that repomirror is taking 5-10 seconds to respond to each request - Log on to winston, notice that repomirror is leaking memory, and eating all the swap - Restart repomirror@winston - Still slow - Profiling says waiting for <https://www.parabola.nu/mirrors/status/tier/1/json/> is taking the time - Log on to proton, see nothing obvious, but things are slow - See most of the swap space belongs to uwsgi processes; restart all uwsgi services on proton, see better iowait / latency - Verify that the mirrors json URL now comes back quickly - repomirror still slow - Mirrors json URL is only slow *from winston*. - ping says connection between winston and proton is plenty fast, but took a while to say it - Verify that DNS lookup of www.parabola.nu is what is slow (winston uses 1984's DNS) - Try to log in to 1984 management panel to see what's up - Can't log in, get "Sorry, but it seems the backend server is feeling sad." from 1984's site. - See in winston's /etc/netctl/... that there are 2 DNS servers configured. Ping them each. - One seems to be totally down, the other seems fine. - Comment out the down one, reload net configuration - Everything seems happy now - Notice pbot is offline, assume bad DNS killed him, restart him
-- Happy hacking, ~ Luke Shumaker
_______________________________________________ Dev mailing list Dev@lists.parabola.nu https://lists.parabola.nu/mailman/listinfo/dev