[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Addshore closed subtask T288534: Add job runner to Wikibase docker-compose example setup & docs as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T255259 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lucas_Werkmeister_WMDE, Addshore Cc:

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Addshore closed this task as "Resolved". Addshore added a comment. > Figure out if numbers greater than 1 even work here So ultimately the answer to this part is yes? However we want to document and provide an example for a job runners for the docker setup? Sounds good to me! TASK DE

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Lucas_Werkmeister_WMDE moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞) board. Lucas_Werkmeister_WMDE added a comment. Alright, it sounds like we have a followup: T288534: Add job runner to Wikibase Docker setup

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Ladsgroup added a comment. I understand but my point is that if you need rate more than 1, you probably need a separate job runner (which can be very easily setup, probably even in docker-compose as part of our official release) TASK DETAIL https://phabricator.wikimedia.org/T255259 EMAIL

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Lucas_Werkmeister_WMDE added a comment. Yes, you’re describing all the downsides of pageview-driven job running, but that doesn’t change the fact that it exists :D TASK DETAIL https://phabricator.wikimedia.org/T255259 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/ema

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Ladsgroup added a comment. That would be weird and interesting and at best would be relying on the webserver's traffic (otherwise, how Apache would know to run the job?). If a server is scaling out that (which I'm surprised even exists), the right way is to have a job runner (as a minutely c

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Lucas_Werkmeister_WMDE added a comment. > in your localhost, no job is being ran unless you run runJobs.php. Right? …no? Jobs are being run, but only on `index.php` requests, not `api.php` ones. But I definitely saw the job queue go down without ever running `runJobs.php`. TASK DETAIL

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Ladsgroup added a comment. It might sound stupid and you probably know this already and I'm really sorry for repeating something you know already but in your localhost, no job is being ran unless you run runJobs.php. Right? If you have some systems in docker to run jobs, you can just spa

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Lucas_Werkmeister_WMDE added a comment. I think adding a job runner to the docker setup, either in `docker-compose.extra.yml` or even in `docker-compose.yml`, would be good, yeah. I briefly looked into it but couldn’t figure out how to attach the MediaWiki/Wikibase config (`LocalSettings.php

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

toan added a comment. In T255259#7266601 , @Lucas_Werkmeister_WMDE wrote: > My assumption was that, with a sufficiently high job run rate, you could run large Wikibase imports, and each edit API request would on average run as many jobs

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Lucas_Werkmeister_WMDE added a comment. I reset my wiki and repeated the import with the job run rate set to 150, and there were still 3003 jobs in the queue after the import was done. Manual page views still seem to chip away at the job queue, in pretty large chunks (e.g. 80 at once), but A

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Lucas_Werkmeister_WMDE added a comment. Okay I bumped the `$wgJobRunRate` to **100**, made another 1000 curl requests (which now took 1m50s instead of 1m20s), and finally the job queue went down significantly, from 2941 to 744 jobs. So with a rate of ostensibly 100 jobs per request, we got j

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Lucas_Werkmeister_WMDE added a comment. I changed the job queue run rate from 2 to 1 in LocalSettings just in case rates > 1 aren’t supported after all, sent 1000 requests to the wiki with curl, and the number of jobs went down from 3010 to… 2941. Is it possible that post-request job run

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Lucas_Werkmeister_WMDE added a comment. It seems pretty clear that jobs were not run during the QuickStatements run – most job rows are still contiguous: name=eval.php > $diffs = []; $prev = 29; foreach ( wfGetDB( DB_REPLICA )->selectFieldValues( 'job', 'job_id', '*', '', [ 'ORDER BY

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Lucas_Werkmeister_WMDE added a comment. In fact, as I go around the wiki, not editing but just loading different pages, the number of jobs tends to go //up//? It fluctuates a bit, sometimes it goes down by one or two, but on the whole it goes up more often, and I’m now at 3009 reported jobs

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Lucas_Werkmeister_WMDE added a comment. On a more-or-less default Wikibase Docker setup (where `$wgJobRunRate` is apparently 2, according to `eval.php`), after creating 1000 items with QuickStatements, the API reports just over 3000 jobs in the queue. However, reloading `api.php` doesn’t

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Lucas_Werkmeister_WMDE claimed this task. Lucas_Werkmeister_WMDE moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞) board. TASK DETAIL https://phabricator.wikimedia.org/T255259 WORKBOARD https://phabricator.wikimedia.

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Lucas_Werkmeister_WMDE added a comment. It looks like `$wgJobRunRate > 1` is actually supported, and has been since the job queue was introduced – looks like this in rMWeeb84cce20e9: Added job table, for deferred processing of jobs. The immediate application is…

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Addshore moved this task from Prioritized Product (prioritised from top to bottom) to Wikidata-Campsite-Iteration-∞ on the Wikidata-Campsite board. Addshore edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞); removed Wikidata-Campsite. TASK DETAIL https://phabricator.wiki

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Addshore updated the task description. Addshore set the point value for this task to "5". TASK DETAIL https://phabricator.wikimedia.org/T255259 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: danshick-wmde, Samantha_Alipio_WMDE, Addshore,

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Samantha_Alipio_WMDE added a project: Wikidata-Campsite. TASK DETAIL https://phabricator.wikimedia.org/T255259 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Samantha_Alipio_WMDE Cc: danshick-wmde, Samantha_Alipio_WMDE, Addshore, Aklapper, Invadibot,

[Wikidata-bugs] [Maniphest] T255259: Investigate increasing the default run rate of jobs for Wikibase

Addshore edited projects, added Wikibase Release Strategy; removed Wikibase-Docker-2017+. Restricted Application added a project: wdwb-tech. TASK DETAIL https://phabricator.wikimedia.org/T255259 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshor