[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.1; 2024-04-16). TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lucas_Werkmeister_WMDE, ReleaseTaggerBot Cc: ArthurTaylor, hoo,

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lucas_Werkmeister_WMDE, Maintenance_bot Cc: ArthurTaylor, hoo, Lucas_Werkmeister_WMDE,

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

gerritbot added a comment. Change #987962 **merged** by jenkins-bot: [mediawiki/extensions/Wikibase@master] Make addUsedEntities aware of statement grouping https://gerrit.wikimedia.org/r/987962 TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE claimed this task. Lucas_Werkmeister_WMDE moved this task from Ready for Peer Review to In Peer Review on the Wikidata Dev Team (Wikidata.org Slice) board. TASK DETAIL https://phabricator.wikimedia.org/T255706 WORKBOARD

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

gerritbot added a comment. Change #989125 **abandoned** by Lucas Werkmeister (WMDE): [mediawiki/extensions/Wikibase@wmf/1.42.0-wmf.12] Add debug code for entity usage logic issue Reason: apparently I didn’t abandon this one, but this train is long gone ^^

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ArthurTaylor removed ArthurTaylor as the assignee of this task. TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ArthurTaylor Cc: ArthurTaylor, hoo, Lucas_Werkmeister_WMDE, ItamarWMDE, Ladsgroup,

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ArthurTaylor claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ArthurTaylor Cc: ArthurTaylor, hoo, Lucas_Werkmeister_WMDE, ItamarWMDE, Ladsgroup, Krinkle, eprodromou, aaron,

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE moved this task from Ready for Tech Verification to Ready for Development on the Wikidata Dev Team (Wikidata.org Slice) board. Lucas_Werkmeister_WMDE added a comment. The deadlock errors have certainly gone down again since the middle of March (when we merged a fix):

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. In T255706#9529038 , @Lucas_Werkmeister_WMDE wrote: > Arthur and I looked at some of the occurrences in Logstash, and two affected pages on itwiki are Ungheria

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE closed subtask T355877: [CLIENT] Investigate potential references to wrong ParserOutput in WikibaseClient code as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. In T255706#9512254 , @ArthurTaylor wrote: > I still have a patch that we haven't applied that makes the `addUsages` job

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. Arthur and I looked at some of the occurrences in Logstash, and two affected pages on itwiki are Ungheria and Ridley Scott ; in both we see the same

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ArthurTaylor added a comment. Looking at Logstash , there is still an issue here, but it comes up much less often now (reduced by around 90%). I still have a patch

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.16; 2024-01-30). TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ReleaseTaggerBot Cc: ArthurTaylor, hoo, Lucas_Werkmeister_WMDE,

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

gerritbot added a comment. Change 992135 **merged** by jenkins-bot: [mediawiki/extensions/Wikibase@master] Fix bug in usage tracking for Parsoid parses https://gerrit.wikimedia.org/r/992135 TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ArthurTaylor removed ArthurTaylor as the assignee of this task. TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ArthurTaylor Cc: ArthurTaylor, hoo, Lucas_Werkmeister_WMDE, ItamarWMDE, Ladsgroup,

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

gerritbot added a comment. Change 992135 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor): [mediawiki/extensions/Wikibase@master] Fix bug in usage tracking for Parsoid parses https://gerrit.wikimedia.org/r/992135 TASK DETAIL

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. In T255706#9473192 , @Lucas_Werkmeister_WMDE wrote: > An alternative might be the `ParserClearState` hook, which is called in (guess what) `Parser::clearState()`. We could perhaps subscribe to

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. Nice investigation, thank you! Now it’s making more sense. > My quick fix involves checking in the singleton getter to see if the `ParserOutput` object reference has changed I think this is actually a pretty good approach – the identity of the

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ArthurTaylor added a comment. So after a lot more staring at stack traces and with some help from base banana , I have a very concrete theory about the cause of the issue, and I am experimentally able to resolve the issue on my local machine. As part of the

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ArthurTaylor added a comment. I did some more digging today. Per the comments from Isabelle in this slack thread , when Parsoid processes Wikitext, it spins up an isolated sub-parse with its own ParserOutput.

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ArthurTaylor claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ArthurTaylor Cc: ArthurTaylor, hoo, Lucas_Werkmeister_WMDE, ItamarWMDE, Ladsgroup, Krinkle, eprodromou, aaron,

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. We deployed PS2 of the debug code on mwdebug2001, and only saw two logstash messages: - T255706 debug: onParserCacheSaveComplete with 47 usages, 79 currentUsages, 0

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. We deployed PS1 of the debug code on mwdebug2001 and null-edited eowiki:Perseo. The result was this logstash event

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. We looked around this a bit more; you can get Parsoid output with useparsoid=1 , and non-Parsoid output with useparsoid=0 (you can see the

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

gerritbot added a comment. Change 984848 **abandoned** by Lucas Werkmeister (WMDE): [mediawiki/extensions/Wikibase@wmf/1.42.0-wmf.9] Add debug code for entity usage logic issue Reason: wmf.9 is no longer deployed https://gerrit.wikimedia.org/r/984848 TASK DETAIL

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

gerritbot added a comment. Change 989125 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)): [mediawiki/extensions/Wikibase@wmf/1.42.0-wmf.12] Add debug code for entity usage logic issue https://gerrit.wikimedia.org/r/989125 TASK

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ArthurTaylor added a comment. In some further investigation today, I was able to reproduce at least part of the issue on my local setup by doing the following, with `$wgWBClientSettings['entityUsageModifierLimits']['C'] = 2` - Start, as expected, with generic C statement in database -

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

gerritbot added a comment. Change 987962 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor): [mediawiki/extensions/Wikibase@master] Make addUsedEntities aware of statement grouping https://gerrit.wikimedia.org/r/987962 TASK DETAIL

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. > without an explicit -o option, it replays the entire message buffer, so we were seeing jobs from December 15: apparently the buffer for this topic spans 7 days. This is actually useful because we can still see the job that was enqueued as a

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. In T255706#9423170 , @Lucas_Werkmeister_WMDE wrote: > We tried various commands along the lines of > > kafkacat -C -b kafka-jumbo1013.eqiad.wmnet:9092 -t

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. In T255706#9422250 , @Krinkle wrote: > In T255706#9415091 , @Lucas_Werkmeister_WMDE wrote: > >> […] I’m also not entirely sure if mwdebug

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ArthurTaylor added a comment. We did a bit more digging this morning. Based on our previous investigation, it seems that we get the same list of statements from the parser cache, with the same (deduplicated) set of usages (i.e. without individual `C:XXX` statements). We had a dig

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Krinkle added a comment. In T255706#9415091 , @Lucas_Werkmeister_WMDE wrote: > […] I’m also not entirely sure if mwdebug ensures that jobs queued by it only run on mwdebug as well; […] Not only do we not ensure this, jobs never run

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. I temporarily deployed this debug change on mwdebug2001 and null-edited eowiki:Perseo there. (Twice, because the first time I didn’t look

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

gerritbot added a comment. Change 984136 **abandoned** by Lucas Werkmeister (WMDE): [mediawiki/extensions/Wikibase@master] Add debug code for entity usage logic issue Reason: this should really have been based on wmf.9 https://gerrit.wikimedia.org/r/984136 TASK DETAIL

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ArthurTaylor added a comment. Update: We (@Lucas_Werkmeister_WMDE, @Michael)have the theory that there are two different parse results with different counts of statements, and that is causing this conflict. We believe the two codepaths are: - doLinksUpdateComplete ->

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

gerritbot added a comment. Change 984848 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)): [mediawiki/extensions/Wikibase@wmf/1.42.0-wmf.9] Add debug code for entity usage logic issue https://gerrit.wikimedia.org/r/984848 TASK DETAIL

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. I put together a few Wikidata queries to show the different properties, as well as whether the properties are present on the Perseus item : - the 52 properties from the fresh parse -

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. I tinkered a bit more with this in production (`mwscript shell eowiki` on mwdebug1001). Specifically, I was curious what the entity usages for eowiki:Perseo looked like before they were deduplicated / combined, so I hacked together some code to render

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. > We were able to observe (in two separate SQL requests) that the `C.XXX` statements were completely absent and then were inserted again Specifically, we saw for **eowiki:Perseo ** (arbitrary test page):

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ArthurTaylor added a comment. We @Michael, @Lucas_Werkmeister_WMDE continued our investigations. There are two apparently related issues in production: - There are pages that have both specific C.XXX statements and a generic C statement. - When we purge these pages

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: ArthurTaylor, hoo, Lucas_Werkmeister_WMDE, ItamarWMDE, Ladsgroup, Krinkle, eprodromou,

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

gerritbot added a comment. Change 984136 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)): [mediawiki/extensions/Wikibase@master] Add debug code for entity usage logic issue https://gerrit.wikimedia.org/r/984136 TASK DETAIL

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a subscriber: ArthurTaylor. Lucas_Werkmeister_WMDE added a comment. @Michael, @ArthurTaylor and I looked into this issue some more and started noticing some things that haven’t been discussed here yet AFAICT: - The queries that deadlock always try to insert a

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Michael moved this task from Unified DOT Backlog to Wikidata.org Slice on the Wikidata Dev Team board. Michael edited projects, added Wikidata Dev Team (Wikidata.org Slice); removed Wikidata Dev Team. TASK DETAIL https://phabricator.wikimedia.org/T255706 WORKBOARD

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. > - remove the waitForReplication between each batch or > - make the job, queue another job instead of doing batches all in one job (the refreshlinks way). I think I’d go for both of these. In `EntityUsageTable`, remove the wait for replication;

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

hashar added a comment. I have removed a log filter from Kibana MediaWiki New Errors dashboard which pointed back to this task. There is no hit over the last eight weeks. The filter was //T255706

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Ladsgroup added a comment. Since jobs also have implicit transaction, the start and end atomic doesn't matter much (I know...) You could potentially: - make a TRX_AUTOCOMMIT connection or - remove the waitForReplication between each batch or - make the job, queue another job

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

Lucas_Werkmeister_WMDE added a comment. In T255706#9231426 , @Lucas_Werkmeister_WMDE wrote: > The waiting was added, with explicit `startAtomic()`+`endAtomic()`, in Make batches in EntityUsageTable::addUsages atomic

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ItamarWMDE renamed this task from "[SW] [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock" to "[WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock". TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ItamarWMDE added a comment. **Task Prio Notes:** - Does affect end users / production - Does not affect monitoring - Does not affect development efforts - Might affect onboarding efforts (if we include documentation improvements as an A/C of this task) - Does affect additional

[Wikidata-bugs] [Maniphest] T255706: [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock

ItamarWMDE renamed this task from "Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock" to "[WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock". TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES