| Gehel added a comment. |
Digging into this a bit more from the WDQS side, we see a few interesting things:
- The NoHttpResponseException seems to not be a timeout client side, but an empty response (not even headers), with a state transition. It looks similar to what we would see if an intermediate proxy would cleanly terminate the TCP connection before the server had sent any data (so timeout on the proxy < timeout on the server)
- We see many more occurrences of the issue on the public wdqs cluster in eqiad than on the others (so more occurrences on the cluster that has more load). See kibana.
- High lag on wdqs (updates from wikidata lagging behind) are sometimes correlated with high rate of NoHttpResponseException. But that correlation is far from perfect, with high rate of NoHttpResponseException happening with no impact on wdqs lag.
- The high lag times are correlated with high system load. This would indicate a correlation between load on blazegraph and timeouts on wikidata. Maybe through CPU starvation. Note that we also see wikidata timeouts when load is low.
I can come up with plenty of scenarios which explain part of the symptoms, but nothing that make sense of all of them. I'm not entirely sure that the issue is on the wikidata side. It is possible that the issue (or at least part of the issue) is related to WDQS itself.
TASK DETAIL
EMAIL PREFERENCES
To: Gehel
Cc: Ladsgroup, Lydia_Pintscher, WMDE-leszek, Aklapper, Gehel, Smalyshev, AndyTan, Davinaclare77, Qtn1293, Imarlier, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Vali.matei, Zppix, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi
Cc: Ladsgroup, Lydia_Pintscher, WMDE-leszek, Aklapper, Gehel, Smalyshev, AndyTan, Davinaclare77, Qtn1293, Imarlier, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Vali.matei, Zppix, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
