[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

gerritbot added a comment. Change 408629 abandoned by Ladsgroup: Add edit and create rate limit for wikidatawiki Reason: Duplicate https://gerrit.wikimedia.org/r/408629TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Lydia_Pintscher added a comment. @Multichill let's look at the remaining issues at the hackathon together. I've put it on my list of things to discuss.TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Multichill added a comment. @Lydia_Pintscher @Ladsgroup any status update on this?TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, MultichillCc: Liuxinyu970226, Legoktm, Lea_Lacroix_WMDE, Dhx1,

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Sjoerddebruin added a comment. Seems like this is affecting the MassMessage extension: https://www.wikidata.org/w/index.php?title=Special:Log/massmessage==250=massmessage=TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Magnus added a comment. @Multichill agreed. The replag solution works AFAICT, why not duplicate that.TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, MagnusCc: Dhx1, Stashbot, Edgars2007,

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Dhx1 added a comment. The main issue I see with the hard limit of 40 creations/minute and 80 edits/minute is that it prevents bots from speeding up during quieter times of each day. RFC 6585 provides the "429 Too Many Requests" HTTP status code (see https://tools.ietf.org/html/rfc6585#page-3)

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Addshore added a comment. In T184948#4139228, @hoo wrote: Why is there a line at 60 edits per minute? Creation is at 40 per minute and edits are at 80 per minute. That would be because I miss read the patch *fixes now*TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

hoo added a comment. In T184948#4139171, @Lucas_Werkmeister_WMDE wrote: Did we announce this limit to the community? I don't think so. In T184948#4139221, @Addshore wrote: I have added some lines to the graph on grafana representing these limits. F17086190: image.png Why is there a line at

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Addshore added a comment. I have added some lines to the graph on grafana representing these limits. F17086190: image.pngTASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, AddshoreCc: Stashbot,

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Lucas_Werkmeister_WMDE added a comment. Did we announce this limit to the community?TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, Lucas_Werkmeister_WMDECc: Stashbot, Edgars2007, Daniel_Mietchen,

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2018-04-18T13:16:53Z] Synchronized wmf-config/InitialiseSettings.php: [[gerrit:427156|Limit page creation and edit rate on Wikidata (T184948)]] (duration: 01m 17s)TASK

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

gerritbot added a comment. Change 427156 merged by jenkins-bot: [operations/mediawiki-config@master] Limit page creation and edit rate on Wikidata https://gerrit.wikimedia.org/r/427156TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

gerritbot added a comment. Change 427156 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani): [operations/mediawiki-config@master] Limit page creation and edit rate on Wikidata https://gerrit.wikimedia.org/r/427156TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Lydia_Pintscher added a comment. @hoo, @Ladsgroup and I sat down together and talked it through. I updated the task description accordingly with what I believe is the minimum requirement to keep the infrastructure sane.TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

hoo added a comment. I just added documentation for the key: https://www.mediawiki.org/w/index.php?diff=2757174=2675575TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: Edgars2007, Daniel_Mietchen,

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

hoo added a comment. Actually, it seems we could also have only specific limits that are not bypassable. While this is not documented (as far as I can tell), one can add '' => false to a specific rate limit action, like the following: > print_r($wgRateLimits); Array ( … [badoath] => Array

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

hoo added a comment. Note: We currently also have dispatch problems while no one is going faster than 75 edits per minute (as far as I can tell)TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc:

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Addshore added a comment. In T184948#4100013, @Magnus wrote: I like the single-shot API request, but additionally I'd like a notification in the edit reply about server-side issues, as is done for replag at the moment. That could save me the additional API query before. Should be possible to to

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Magnus added a comment. @Addshore It depends on what the actual issue with creating/editing is. If it's database-related (most likely), then having a few sleeping processes for load-dependent server-side delays might be a viable solution. I like the single-shot API request, but additionally I'd

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

ArtixKreiger added a comment. So can i get back to running QuickStatement batches?TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArtixKreigerCc: Edgars2007, Daniel_Mietchen, Pasleim, Magnus, abian,

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Multichill added a comment. In T184948#4089557, @Magnus wrote: So, either WMF expose the "overload status" so we can throttle on demand, or WMF delay the API response accordingly, so we don't have to do anything, or WMF scale to demand :-) Yep. WMF or WMDE I guess. @Lydia_Pintscher can you

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Magnus added a comment. So, either WMF expose the "overload status" so we can throttle on demand, or WMF delay the API response accordingly, so we don't have to do anything, or WMF scale to demand :-) TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Multichill added a comment. Hi Magnus, In T184948#4087139, @Magnus wrote: That is a total of three different rate limitations (two dynamic, one hardcoded fallback). Personally, I consider that beyond due diligence, verging on paranoia. Therefore, I consider this matter resolved, as far as my

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Magnus added a comment. OK, let me summarize my situation, as of today: Essentially, all my Wikidata edits (both bots and _javascript_ on Toolforge) run through the same OAuth PHP class I wrote Every single edit (including "create item") runs with a "maxlag=5" parameter. If the edit fails

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

ArtixKreiger added a comment. this guy artix waits for the rate limit to implemented so he can go back and finish is batch job.TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArtixKreigerCc: Edgars2007,

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Daniel_Mietchen added a comment. I like the suggestion of adding a load-dependent server-side delay for API edits.TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Daniel_MietchenCc: Daniel_Mietchen, Pasleim,

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Multichill added a comment. In T184948#4079500, @Magnus wrote: I have added a 5 second delay after item creations. I also added a mechanism to have a delay after any other edit type. Note that this limits single threads only. QuickStatements, various SourceMD webtools and bots, Mix'n'match sync

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Magnus added a comment. For one tool, per user, per tab? Sure. I'd just set both item creation and edit to 1 second delay. For one tool, all users? No. At least, not easily. Question: WMF knows best when their servers are overloaded. Why not just add a delay for the API response when the servers

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

ArtixKreiger added a comment. Magnus, so what about the edit rate limit of 100/min? Is it hard coded?TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArtixKreigerCc: Magnus, abian, Lucas_Werkmeister_WMDE,

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Magnus added a comment. I have added a 5 second delay after item creations. I also added a mechanism to have a delay after any other edit type. Note that this limits single threads only. QuickStatements, various SourceMD webtools and bots, Mix'n'match sync etc. can still add up. Also, this does

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Magnus added a comment. Thanks to @Multichill for pointing out this discussion to me... As for [[Bus factor]], I have always welcomes co-maintainers on my tools, and have repeatedly suggested WMF involvement, anywhere from taking over/rewriting to just becoming co-maintainers of important tools.

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

ArtixKreiger added a comment. Addshore, it seems Magnus is the only one who maintains the tool. If he feels like sleeping or do something else in his life, there would be a big set of shoes to fix, and not an easy pair of shoes to fix.TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Multichill added a comment. In T184948#4078538, @Lydia_Pintscher wrote: There are other tools that are going too fast like fatameh as well. This isn't something we should have to correct on a tool-by-tool basis. That's what you getting from being a bit more liberal for a long time. Tool rights

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Lydia_Pintscher added a comment. There are other tools that are going too fast like fatameh as well. This isn't something we should have to correct on a tool-by-tool basis.TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Multichill added a comment. In T184948#4075653, @Ladsgroup wrote: In T184948#4059447, @Multichill wrote: In T184948#4052662, @Ladsgroup wrote: I personally would like to strip the right from the bots and enforce a rate limit for them as they just don't care about what we say. If bots ignore

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Ladsgroup added a comment. In T184948#4059447, @Multichill wrote: In T184948#4052662, @Ladsgroup wrote: I personally would like to strip the right from the bots and enforce a rate limit for them as they just don't care about what we say. If bots ignore replag (

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Addshore added a comment. In T184948#4064358, @ArtixKreiger wrote: Whats the point of noratelimit if admins and bots are rate limited? We can always have multiple limits. Also noratelimit is part of mediawiki, we didn't create it for wikibase, but we are now finding out that perhaps it is a bad

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Addshore added a comment. In T184948#4059447, @Multichill wrote: If bots ignore replag ( https://meta.wikimedia.org/wiki/Bot_policy#Edit_throttle_and_peak_hours ) these bots should just be stripped of the botflag. The good shouldn't suffer because of a small number of problematic users. The

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

ArtixKreiger added a comment. Whats the point of noratelimit if admins and bots are rate limited? However, yes. I agree with everyone that there should be some controllable edit rate. This is a random suggestion, but feel free to ignore. Perhaps, for "everyone", there is an enforced 100/min edit

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Multichill added a comment. In T184948#4052662, @Ladsgroup wrote: I personally would like to strip the right from the bots and enforce a rate limit for them as they just don't care about what we say. If bots ignore replag ( https://meta.wikimedia.org/wiki/Bot_policy#Edit_throttle_and_peak_hours

[Wikidata-bugs] [Maniphest] [Commented On] T184948: limit page creation and edit rate on Wikidata

Ladsgroup added a comment. I personally would like to strip the right from the bots and enforce a rate limit for them as they just don't care about what we say.TASK DETAILhttps://phabricator.wikimedia.org/T184948EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: