[Wikidata-bugs] [Maniphest] [Commented On] T137539: Use individual select queries to determin which Entities are in the wbc_entity_usage table
gerritbot added a comment.Change 293745 merged by jenkins-bot: EntityUsageTable: Run all subqueries at once with UNION on MySQL https://gerrit.wikimedia.org/r/293745TASK DETAILhttps://phabricator.wikimedia.org/T137539EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Zppix, Aklapper, hoo, aude, aaron, daniel, jcrespo, Vali.matei, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, GWicke, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T137539: Use individual select queries to determin which Entities are in the wbc_entity_usage table
jcrespo added a comment.I like your approach @daniel, starting simple, then optimize with a more complex option. Thank you!TASK DETAILhttps://phabricator.wikimedia.org/T137539EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc: gerritbot, Zppix, Aklapper, hoo, aude, aaron, daniel, jcrespo, Vali.matei, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, GWicke, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T137539: Use individual select queries to determin which Entities are in the wbc_entity_usage table
daniel added a comment.Note: the naive "one query per ID" patch (293736) is merged, and should already fix the performance issue. The "use a UNION" patch (293745) is still pending comments and testing. The second patch isn't strictly necessary, but provides another 10x speedup (for large sets of IDs), according to Marius' tests. I'd leave this task open until the second patch is either merged or abandoned.TASK DETAILhttps://phabricator.wikimedia.org/T137539EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, danielCc: gerritbot, Zppix, Aklapper, hoo, aude, aaron, daniel, jcrespo, Vali.matei, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, GWicke, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T137539: Use individual select queries to determin which Entities are in the wbc_entity_usage table
gerritbot added a comment.Change 293736 merged by jenkins-bot: Use one query per entity id in getUsedEntityIdStrings https://gerrit.wikimedia.org/r/293736TASK DETAILhttps://phabricator.wikimedia.org/T137539EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Zppix, Aklapper, hoo, aude, aaron, daniel, jcrespo, Vali.matei, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, GWicke, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T137539: Use individual select queries to determin which Entities are in the wbc_entity_usage table
gerritbot added a comment.Change 293745 had a related patch set uploaded (by Hoo man): EntityUsageTable: Run all subqueries at once with UNION on MySQL https://gerrit.wikimedia.org/r/293745TASK DETAILhttps://phabricator.wikimedia.org/T137539EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Zppix, Aklapper, hoo, aude, aaron, daniel, jcrespo, Vali.matei, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, GWicke, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T137539: Use individual select queries to determin which Entities are in the wbc_entity_usage table
hoo added a comment.Short test:
hoo@terbium:~$ mwscript eval.php --wiki=zhwiki
> $idStrings = [];
> for ( $i = 1; $i < 1001; $i++ ) { $idStrings[] = "Q$i"; }
> $subQueries = [];
> $connection = wfGetDB( DB_SLAVE );
> foreach ( $idStrings as $idString ) { $subQueries[] = $connection->selectSQLText( 'wbc_entity_usage', 'eu_entity_id', [ 'eu_entity_id' => $idString ], '', [ 'LIMIT' => 1 ] ); }
> $sql = $connection->unionQueries( $subQueries, false );
> $t0 = microtime( true ); $res = $connection->query( $sql ); echo microtime( true ) - $t0;
0.10685896873474TASK DETAILhttps://phabricator.wikimedia.org/T137539EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: gerritbot, Zppix, Aklapper, hoo, aude, aaron, daniel, jcrespo, Vali.matei, Lewizho99, Maathavan, D3r1ck01, Izno, Wikidata-bugs, GWicke, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T137539: Use individual select queries to determin which Entities are in the wbc_entity_usage table
gerritbot added a comment.Change 293736 had a related patch set uploaded (by Hoo man): WIP: Use one (sub)query per entity id in getUsedEntityIdStrings https://gerrit.wikimedia.org/r/293736TASK DETAILhttps://phabricator.wikimedia.org/T137539EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: gerritbot, Zppix, Aklapper, hoo, aude, aaron, daniel, jcrespo, Vali.matei, D3r1ck01, Izno, Wikidata-bugs, GWicke, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
