Lydia_Pintscher added a comment.
I'll ask people to have another look here,TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lydia_PintscherCc: Agabi10, Lucas_Werkmeister_WMDE, gerritbot, Addshore,
Emijrp added a comment.
Editing Wikidata is pretty slow for me now.TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lydia_Pintscher, EmijrpCc: Agabi10, Lucas_Werkmeister_WMDE, gerritbot, Addshore,
Bugreporter added a comment.
I don't think this is resolved, see https://grafana.wikimedia.org/dashboard/db/wikidata-dispatch?refresh=1m=1=now-90d=now
The median dispatch lag is almost never higher than 200/20s before 2017-06-29 (and since we start to record this), but is always higher than 1
Esc3300 added a comment.
Good news!
I wonder if we now agree what people should look for and what should be done if there are delays.TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lydia_Pintscher,
Ladsgroup added a comment.
Now the stalest wiki is 3 minutes, should we close this?TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: LadsgroupCc: Agabi10, Lucas_Werkmeister_WMDE, gerritbot, Addshore,
gerritbot added a comment.
Change 370315 merged by Jcrespo:
[operations/puppet@production] mediawiki: Another increase of batch size in dispatchChanges cronjob
https://gerrit.wikimedia.org/r/370315TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL
Sjoerddebruin added a comment.
The dispatch decreased a lot since yesterday, but is increasing again since botimport for cebwiki and svwiki started again. I hope the above patch helps with keeping the dispatch steady.TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL
gerritbot added a comment.
Change 370315 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] mediawiki: Another increase of batch size in dispatchChanges cronjob
https://gerrit.wikimedia.org/r/370315TASK
daniel added a comment.
In T171263#3497960, @Esc3300 wrote:
Is there a way to meter specifically the update channel for enwiki?
Not really - there is one queue for dispatching to enwiki, one queue for receiving on enwiki, and then there are several queues for processing different kinds of
Esc3300 added a comment.
Is there a way to meter specifically the update channel for enwiki?
e.g. (wikidata edit > queue 1 > queue 2 > queue 3 > .. > update enwiki : actual size / delay )TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL
daniel added a comment.
Quick status summary:
Dispatcher batch size has been increased. Seems to have the desired effect, dispatch lag is going down, but only slowly. Maybe we can bump the batch size some more?
Patches for improving throughput on the receiving end have been merged. This has no
gerritbot added a comment.
Change 366887 merged by Filippo Giunchedi:
[operations/puppet@production] mediawiki: increase the batch size of dispatchChanges cronjob
https://gerrit.wikimedia.org/r/366887TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL
Esc3300 added a comment.
In T171263#3474950, @daniel wrote:
Is there much demand for the recent changes feed in client wikis (other than update of displayed statements)? Personally, I find it hard to read, even for my own edits.
I would say yes, as it was considered a precondition to allowing
Esc3300 added a comment.
Changes to descriptions seem lower priority than changes to labels/statements.TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Esc3300Cc: Lucas_Werkmeister_WMDE, gerritbot, Addshore,
Emijrp added a comment.
In T171263#3484037, @daniel wrote:
@Emijrp the point is that at least some people should have an eye on bot edits some time, otherwise nobody will notice when a bot goes wrong. Bot edits on wikidata should be marked as such on client wikis too (let me know if they are
daniel added a comment.
In T171263#3482050, @Lucas_Werkmeister_WMDE wrote:
especially when the bot updates the description in twenty separate edits.
Is it true that this makes a big difference? Aren’t changes batched together?
Changes get batched (coalesced) together on the client side, not
daniel added a comment.
@Emijrp the point is that at least some people should have an eye on bot edits some time, otherwise nobody will notice when a bot goes wrong. Bot edits on wikidata should be marked as such on client wikis too (let me know if they are not), so people can filter them out. But
Emijrp added a comment.
In T171263#3460518, @daniel wrote:
To avoid confusion: dispatch lag is the time that changes sit around before they go into the client wikis' job queue. The time they spent in the job queue is not the issue here! Changes "sit around" because finding the changes relevant
Emijrp added a comment.
I think that the current dispatchlag growing is due to ResearchBot academic paper item creation. I say that because some days ago my bot was editing at 60 ed/min and ResearchBot was doing a similar rate. I stopped my bot for a few hours and the dispatchlag kept growing. So
Lucas_Werkmeister_WMDE added a comment.
especially when the bot updates the description in twenty separate edits.
Is it true that this makes a big difference? Aren’t changes batched together?TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL
Esc3300 added a comment.
In T171263#3481214, @Emijrp wrote:
And what is causing this? Bot page creation or bot edits?
From Daniel's explanation and the documentation he provided, it seems that the more pages are linked to an item, the larger the impact ..
If page creations are only for
Emijrp added a comment.
In T171263#3480923, @Lucas_Werkmeister_WMDE wrote:
Stalest wiki now more than one day behind again. Median is fine, but as the stalest wiki is enwiki, that’s not a great consolation.
And what is causing this? Bot page creation or bot edits?TASK
Lucas_Werkmeister_WMDE added a comment.
Stalest wiki now more than one day behind again. Median is fine, but as the stalest wiki is enwiki, that’s not a great consolation.TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL
daniel added a comment.
@Esc3300 https://phabricator.wikimedia.org/diffusion/EWBA/browse/master/docs/change-propagation.wikiTASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: danielCc: Lucas_Werkmeister_WMDE,
Esc3300 added a comment.
Thanks for your response. Helped me understand how the pipeline(s) work.TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Esc3300Cc: Lucas_Werkmeister_WMDE, gerritbot, Addshore,
daniel added a comment.
In T171263#3474757, @Esc3300 wrote:
Could the situation be improved by limiting the type of changes that are dispatched to various wikis?
We only dispatch changes to wikis that use the given item. Further filtering, as suggested below, happens on the client side. We
Esc3300 added a comment.
Could the situation be improved by limiting the type of changes that are dispatched to various wikis?
I noticed that in some wikis, en labels are systematically subscribed to, but not displayed (possibly some inefficiency in their Module:Wikidata .
Some bots just "update
Sjoerddebruin added a comment.
Still interesting spikes happening. The stalest wiki is wikidatawiki for some reason, is that normal?TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SjoerddebruinCc:
Sjoerddebruin added a comment.
More good progress in the last 24 hour. Will notify the community when things are really down, then we need to keep watching if the dispatch increases too fast again. Editing speed of users is visible here.TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL
Ladsgroup added a comment.
It's not possible to do dispatching on one wiki only but my patch increases the maximum time of each dispatching job, it means two things: 1- We can have four instances of dispatchers instead of three, it's good 2- If dispatching to big wikis time out because of it's too
gerritbot added a comment.
Change 366887 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] mediawiki: increase the maximum time of dispatchChanges cronjob
https://gerrit.wikimedia.org/r/366887TASK
Ladsgroup added a comment.
I'm so happy to see tons of donors money on bandwidth, database storage, and software engineer time are being wasted on articles that never will be read.
Just note that median of dispatch lag is not horrible (2 minutes) and only English Wikipedia and cebwiki are lagging
Addshore added a comment.
I wouldn't say it makes dispatching slower, just there is more to dispatch, and currently the dispatch system can only handle so many changes per second / minute / hour.TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL
matej_suchanek added a comment.
The lag started to increase around 30th July - around that day, the great import of cebwiki items started as well. Now look at when the sharp increase of items ends and when the increase in lag ends. The difference is 4 days, which is the number on the peak of the
daniel added a comment.
@MisterSynergy I agree that a software or config change is a prime candidate for causing an issue like this. But as far as I can tell from https://wikitech.wikimedia.org/wiki/Server_Admin_Log#2017-06-28 there were no relevant changes deployed on 2017-06-28.TASK
MisterSynergy added a comment.
How much are we sure that this problem was predominantly caused by the recently “high” edit rate (or undesired number of parallel bot runs) at Wikidata? At Wikidata we are trying to get edit rates down, but I am uncomfortable with the notion that Wikidata operates
daniel added a comment.
To avoid confusion: dispatch lag is the time that changes sit around before they go into the client wikis' job queue. The time they spent in the job queue is not the issue here!
In any case: one thing we could do is skip old changes. Changes older than a day are unlikely
XXN added a comment.
The core problem is the fact that Wikidata is poorly organized as a community and project. Per overall a big part of the work done is inefficient and counterproductive. At least the descriptioning part could and should be optimized. There are many users who acts just like
PokestarFan added a comment.
As more data gets added, job queue is going to get even higher. More and more bots are going to be added. The day that wiktionary words are added to items, Wikidata is going to overflow.TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL
Sjoerddebruin added a comment.
We indeed lowered the amount of edits to at least try to make the process go faster. Let's hope this is a wakeup call for all of us.TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL
Ladsgroup added a comment.
My bot has 25 million edits in Wikidata, I know it's large. The problem is that we ran too fast for a while and now the backlog is so big that takes some time for the infra to handle to backlog. We should have more resources for Wikidata but resources alone can't fix
Emijrp added a comment.
In T171263#3459885, @Ladsgroup wrote:
1 million queued jobs is absolutely normal. Problem lies somewhere else.
It is 1 million because we have shut down almost all bots. It was 3 million a few days ago.[1]
If you check Wikidata edits Graphana, you will see how Wikidata
Ladsgroup added a comment.
1 million queued jobs is absolutely normal. Problem lies somewhere else.TASK DETAILhttps://phabricator.wikimedia.org/T171263EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: LadsgroupCc: Ladsgroup, Lydia_Pintscher, hoo, Bugreporter,
Emijrp added a comment.
I don't know how many server resources are dedicated to Wikidata (CPU, memory, etc), but could be possible to increase them? Setting another job server?
In the future (maybe a year?) more data will be integrated into Wikidata (Commons structured data and Wiktionary
44 matches
Mail list logo
