[Wikidata-bugs] [Maniphest] [Commented On] T147748: Large number of CategoryMembershipChangeJob::run updates are failing

gerritbot added a comment. Change 315616 merged by jenkins-bot: Use one query per term type in SqlEntityInfoBuilder::collectTermsForEntities https://gerrit.wikimedia.org/r/315616TASK DETAILhttps://phabricator.wikimedia.org/T147748EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T147748: Large number of CategoryMembershipChangeJob::run updates are failing

thiemowmde added a comment. I'm moving this to "done" in our Sprint board, but I will leave this ticket open because I do not know if the only patch mentioned here (https://gerrit.wikimedia.org/r/315616) fully resolves the issue. Please have a look.TASK

[Wikidata-bugs] [Maniphest] [Commented On] T147748: Large number of CategoryMembershipChangeJob::run updates are failing

hoo added a comment. In T147748#2712102, @jcrespo wrote: You could even merge both into UNION ALLs, but again, as usual, I suggest going for the largest gain ASAP. Sadly this is still a major headache in tests because we use temporary tables there… which makes MySQL sad.TASK

[Wikidata-bugs] [Maniphest] [Commented On] T147748: Large number of CategoryMembershipChangeJob::run updates are failing

jcrespo added a comment. I conclude that we should run one query per term type here and simply merge the result in the application code. Yes, this is a property of Btree indexes- 2 ranges cannot be handled efficiently- you can only handle one with an index at a time. You could even merge both

[Wikidata-bugs] [Maniphest] [Commented On] T147748: Large number of CategoryMembershipChangeJob::run updates are failing

gerritbot added a comment. Change 315616 had a related patch set uploaded (by Hoo man): Use one query per term type in SqlEntityInfoBuilder::collectTermsForEntities https://gerrit.wikimedia.org/r/315616TASK DETAILhttps://phabricator.wikimedia.org/T147748EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T147748: Large number of CategoryMembershipChangeJob::run updates are failing

hoo added a comment. I've benchmarked the above query very briefly (picked db1092 for that, all queries were run five times in a row(!)). Original query SELECT /* Wikibase\Lib\Store\Sql\SqlEntityInfoBuilder::collectTermsForEntities */

[Wikidata-bugs] [Maniphest] [Commented On] T147748: Large number of CategoryMembershipChangeJob::run updates are failing

hoo added a comment. In T147748#2704018, @jcrespo wrote: Is the slow Wikibase\Lib\Store\Sql\SqlEntityInfoBuilder::collectTermsForEntities related to the job or is it a separate issue? Separate issue… I already looked into this briefly before and think I have a potential fix. Will probably pick

[Wikidata-bugs] [Maniphest] [Commented On] T147748: Large number of CategoryMembershipChangeJob::run updates are failing

jcrespo added a comment. Thank you for the quick response, seeing the effect already, I am more than ok with putting this as high or normal (sorry, I didn't see it soon). Is the slow Wikibase\Lib\Store\Sql\SqlEntityInfoBuilder::collectTermsForEntities related to the job or is it a separate

[Wikidata-bugs] [Maniphest] [Commented On] T147748: Large number of CategoryMembershipChangeJob::run updates are failing

hoo added a comment. The change has been deployed and the number of Jobs scheduled on Wikidata should drop significantly. It will take some time before the existing queue has dried off, though. You might want to lower the priority to HIGH.TASK DETAILhttps://phabricator.wikimedia.org/T147748EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T147748: Large number of CategoryMembershipChangeJob::run updates are failing

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2016-10-10T12:06:49Z] Synchronized php-1.28.0-wmf.21/extensions/Wikidata: Update Wikibase, add EntityHandler::supportsCategories (T147748) (duration: 02m 25s)TASK DETAILhttps://phabricator.wikimedia.org/T147748EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T147748: Large number of CategoryMembershipChangeJob::run updates are failing

jcrespo added a comment. It is clear to me that the direct cause for connection issues is pilups coming from queries such as: Id: 6451784424 User: wikiuser Host: 10.64.32.35:38688 db: wikidatawiki Command: Query Time: 18 State: Sending data Info: SELECT /*

[Wikidata-bugs] [Maniphest] [Commented On] T147748: Large number of CategoryMembershipChangeJob::run updates are failing

jcrespo added a comment. Probably related, queries like this are causing thousands of errors a day: Hits Tmax Tavg Tsum Hosts Users Schemas 11910 33 4 52,440 db1087 wikiuser wikidatawiki SELECT /* Wikibase\Lib\Store\Sql\SqlEntityInfoBuilder::collectTermsForEntities */ term_entity_type,