| Imarlier added a comment. |
@BBlack @ema Couple of questions for you about Nginx:
- Do we have nginx configured to handle a specific number of requests on a given worker process/thread, and then shut that down?
- If it possible for nginx to be restarted (interrupting existing persistent connections) due to config updates or the like, and if so, is there a record of times when that has happened?
What I'm poking at is the issue described here: https://williamsbdev.com/posts/no-http-response-exceptions/.
The Wikidata updater process has a pool of connections to the text cache nodes, eg
imarlier@wdqs1003:~$ sudo netstat -anet | grep 208.80.154.224 tcp6 0 0 10.64.0.14:50536 208.80.154.224:443 ESTABLISHED 499 283749063 tcp6 0 0 10.64.0.14:50278 208.80.154.224:443 ESTABLISHED 499 283748769 tcp6 0 0 10.64.0.14:51154 208.80.154.224:443 ESTABLISHED 499 283750591 tcp6 0 0 10.64.0.14:50566 208.80.154.224:443 ESTABLISHED 499 283733664 tcp6 0 0 10.64.0.14:50564 208.80.154.224:443 ESTABLISHED 499 283749076 tcp6 0 0 10.64.0.14:51184 208.80.154.224:443 ESTABLISHED 499 283738772 tcp6 0 0 10.64.0.14:51648 208.80.154.224:443 ESTABLISHED 499 283750982 tcp6 0 0 10.64.0.14:50794 208.80.154.224:443 ESTABLISHED 499 283749269 tcp6 0 0 10.64.0.14:51662 208.80.154.224:443 ESTABLISHED 499 283750987 tcp6 0 0 10.64.0.14:50796 208.80.154.224:443 ESTABLISHED 499 283738743
The implication of the article linked above is that if the process that represents the remote side of these connections were to shut down while the connection itself is idle, it's possible that the socket would enter a CLOSE_WAIT state on the local side, and that in turn would result in the NoHttpResponseException that we're seeing in this case.
I'm doing some super super naive testing of this hypothesis on one of the wdqs hosts, but I'm curious whether it strikes you as being possible based on what we see here.
Cc: BBlack, ema, Gehel, Aklapper, Smalyshev, Legado_Shulgin, Nandana, thifranc, AndyTan, Davinaclare77, Qtn1293, Imarlier, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Vali.matei, Zppix, _jensen, D3r1ck01, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
