[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Addshore added a comment.
So the spike reported again above was before https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikibaseQualityConstraints/+/464812/ was merged and deployed.
WBQC now respects the 429 header and will throttle based on it.
I added a new panel to https://grafana.wikimedia.org/d/00344/wikidata-quality so that we can see this along side the http responses from the query service for WBQC.
I also created https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikibaseQualityConstraints/+/483112/ and then abandoned it (to split this metric between pre request and post request), i'm not sure if we need this as we can already calculate this based on # of throttle attempts - # requests that result in 429 = # of requests that were totally avoided due to throttling.
While digging into this again I went to look at how the throttling was performing and found that there were some occurrences of "Sparql API replied with status 429" in the log,
but they don't seem to have a valid retry-after header.
It doesn't look like the query service is returning a valid retry-after header?
So right now WBQC will be just waiting for 10 seconds per the default setting of the WBQualityConstraintsSparqlThrottlingFallbackDuration
An example log entry can be seen @ https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.12.30/mediawiki?id=AWgAckPBzpjgITg61Cm8&_g=h@4e853bb
which says the headers were:
{"server":["nginx\/1.13.6"],"date":["Sun, 30 Dec 2018 18:48:34 GMT"],"content-type":["application\/octet-stream"],"connection":["close"],"cache-control":["no-cache"]}
The WBQC code is specifically looking for 'Retry-After' currently case sensitive.TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: gerritbot, Tpt, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, Gaboe420, Zylc, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, 1978Gage2001, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Chicocvenancio, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Tbscho, merbst, LawExplorer, Lewizho99, Zppix, JJMC89, Maathavan, _jensen, Agabi10, D3r1ck01, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, Lydia_Pintscher, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
gerritbot added a comment. Change 483112 abandoned by Addshore: Split SPARQL pre and post throttling metrics Reason: This can probably be done with the metrics we already have.. https://gerrit.wikimedia.org/r/483112TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Tpt, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, Legado_Shulgin, CucyNoiD, Nandana, NebulousIris, thifranc, AndyTan, Gaboe420, Zylc, Versusxo, Majesticalreaper22, Giuliamocci, Davinaclare77, Adrian1985, Qtn1293, Cpaulf30, 1978Gage2001, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Chicocvenancio, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Tbscho, merbst, LawExplorer, Lewizho99, Zppix, JJMC89, Maathavan, _jensen, Agabi10, D3r1ck01, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, Lydia_Pintscher, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
gerritbot added a comment. Change 483112 had a related patch set uploaded (by Addshore; owner: Addshore): [mediawiki/extensions/WikibaseQualityConstraints@master] Split SPARQL pre and post throttling metrics https://gerrit.wikimedia.org/r/483112TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Tpt, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, Legado_Shulgin, Nandana, thifranc, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, _jensen, Agabi10, D3r1ck01, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, Lydia_Pintscher, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Addshore added a comment. In T204267#4652483, @Smalyshev wrote: Happened again, bumping the priority. The spike can be seen here https://grafana.wikimedia.org/d/00344/wikidata-quality?orgId=1=1539267066438=1539596899898 It doesn't look like there was a bit spike in hits to the api or special pages. There was a big spike in the regex cache misses here.TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Tpt, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, Legado_Shulgin, Nandana, thifranc, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, _jensen, Agabi10, D3r1ck01, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, Lydia_Pintscher, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Smalyshev added a comment. Also, 1,182,961 events is a lot. What's going on there? Why so many? Is it a legit scenario? I wonder also if the most of it is regex matching shouldn't we make some service to do just that. Using full-blown SPARQL database to do regex matching is kinda like hammering in nails using a last-generation cell phone.TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Tpt, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, Nandana, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Smalyshev added a comment. Is there a reason that all mediawiki hosts show as "localhost"? This is probably coming from Jetty, which takes it from connection info. Since we have nginx in front of the Blazegraph and no X-Client-IP is supplied, probably, it has no way to discover the originating host.TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Tpt, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, Nandana, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Addshore added a comment. It looks like there was another little flood on the 1st of October with requests being banned again: https://logstash.wikimedia.org/goto/77f3d01f6e7eaf56e5436727b5643ba2 Is there a reason that all mediawiki hosts show as "localhost"?TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Tpt, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, Nandana, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Addshore added a comment. In T204267#4584397, @Smalyshev wrote: All bans are temporary, so as soon as traffic returns to normal the bans will expire. It would be nice if there was a way to wbqc to respect the 429 throttling header, which will avoid bans. That sounds like a great idea that could do with some thought, I'll file a ticket.TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Smalyshev, AddshoreCc: Tpt, aborrero, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Smalyshev added a comment. All bans are temporary, so as soon as traffic returns to normal the bans will expire. It would be nice if there was a way to wbqc to respect the 429 throttling header, which will avoid bans.TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Tpt, aborrero, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Tpt added a comment. @Jonas Thank you for your feedback. is it necessary for your tool to run the constraint checks in parallel? No, I am going to switch to a sequential processing. Thanks for the idea! Using WDQS instead would be a good idea, because then only your tool would get throttled. It's what I already do to start filling my database. And I was using the recent changes to get the "most recent" violations i.e. the ones that the users are probably the most willing to fix and that have an higher probability to be vandalism. I just restarted the tool and I now only use WDQS and some calls to the QualityApi to get the conflict labels, but only at a 1/s rate.TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Smalyshev, TptCc: Tpt, aborrero, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Jonas added a comment. @Tpt is it necessary for your tool to run the constraint checks in parallel? Using WDQS instead would be a good idea, because then only your tool would get throttled. The problem there is that just a fraction of all violations are in there at the moment.TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Smalyshev, JonasCc: Tpt, aborrero, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Tpt added a comment. Sorry everyone for the troubles. I was experimenting with a tool that tries to find corrections for constraint violations. I have modified it to send a proper User-Agent for all its requests to the Wikidata API but not restarted it. @Wikidata team: what would be an ok request rate to the constraints violation API? The alternative options for me from using the WikibaseQuality API does not look very compelling: look directly for violations from WDQS but it's probably not much better in term of load for the query service and it won't use any cache implemented by WikibaseQuality. use its own WDQS but it means to find an other hosting than tools labs. TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Smalyshev, TptCc: Tpt, aborrero, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Stashbot added a comment. Mentioned in SAL (#wikimedia-cloud) [2018-09-14T11:22:34Z] T204267 stop the corhist tool (k8s) because is hammering the wikidata APITASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, StashbotCc: aborrero, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
aborrero added a comment. In T204267#4583629, @Pintoch wrote: @aborrero thanks for the ping. I do not recognize the shape of the queries as coming from this tool though. The openrefine-wikidata tool should do relatively few SPARQL queries, whose results are cached in redis. How did you determine that this tool is the source of the problem? Sorry for the noise, after stopping your tool we still see the same traffic. Feel free to restart your tool and again: sorry. Problem is that we can't identify the tool by the server logs because there isn't any meaningful UA, so we are walking a bit blind. Your tool had the wikidata name in it, so that was the first candidate.TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, aborreroCc: aborrero, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Pintoch added a comment. @aborrero thanks for the ping. I do not recognize the shape of the queries as coming from this tool though. The openrefine-wikidata tool should do relatively few SPARQL queries, whose results are cached in redis. How did you determine that this tool is the source of the problem?TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, PintochCc: aborrero, Pintoch, Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Stashbot added a comment. Mentioned in SAL (#wikimedia-cloud) [2018-09-14T10:51:35Z] T204267 stop the openrefine-wikidata tool (k8s) because is hammering the wikidata APITASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, StashbotCc: Stashbot, Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, AndyTan, Zylc, Davinaclare77, Qtn1293, 1978Gage2001, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, Th3d3v1ls, Hfbn0, QZanden, EBjune, Tbscho, merbst, LawExplorer, Zppix, JJMC89, Agabi10, Xmlizer, srodlund, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Gryllida, faidon, scfc, Mbch331, Jay8g, Krenair, fgiunchedi, chasemp___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Addshore added a comment. Looks like it is tools-worker-1021.tools.eqiad.wmflabsTASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: AddshoreCc: Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, Agabi10, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Jonas added a comment. Seems the #wikibase-quality-constraints extension is the source: https://grafana.wikimedia.org/dashboard/db/wikidata-quality?refresh=10s=1=now-30d=now https://github.com/wikimedia/mediawiki-extensions-WikibaseQualityConstraints/blob/c3687c1626323134e546ba924f3a838470c1f188/src/ConstraintCheck/Helper/SparqlHelper.php#L486 It seems SPARQL error codes increased drastically F25847853: image.pngTASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: JonasCc: Jonas, Addshore, TerraCodes, Liuxinyu970226, Aklapper, Smalyshev, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Agabi10, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T204267: Flood of WDQS requests from wbqc
Smalyshev added a comment.
Kibana log for banned requests.
Example request:
PREFIX wd:
PREFIX wds:
PREFIX wdt:
PREFIX wdv:
PREFIX p:
PREFIX ps:
PREFIX pq:
PREFIX pqv:
PREFIX pr:
PREFIX prv:
PREFIX wikibase:
PREFIX wikibase-beta: SELECT (REGEX("533892", "^(?:[1-9][0-9]+|)$") AS ?matches) {}=json=5000
Also I must notice about 90% of the text of this request is prefixes which are already defined by default...TASK DETAILhttps://phabricator.wikimedia.org/T204267EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Aklapper, Smalyshev, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Agabi10, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
