hoo added a comment.
Not yet done:
mysql:wikiadmin@db1070 [wikidatawiki]> SELECT COUNT(*) FROM wb_terms WHERE term_full_entity_id IS NULL;
+--+
| COUNT(*) |
+--+
| 674 |
+--+
1 row in set (0.00 sec)TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
gerritbot added a comment.
Change 381421 merged by Jcrespo:
[operations/puppet@production] mediawiki: stop rebuilding wb_terms table
https://gerrit.wikimedia.org/r/381421TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
gerritbot added a comment.
Change 381421 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] mediawiki: stop rebuilding wb_terms table
https://gerrit.wikimedia.org/r/381421TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
gerritbot added a comment.
Change 375741 merged by Jcrespo:
[operations/puppet@production] mediawiki: make the wikidata wb_terms rebuild a little bit faster
https://gerrit.wikimedia.org/r/375741TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
gerritbot added a comment.
Change 375352 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Fix minor issues recently introduced in TermSqlIndexBuilder and test
https://gerrit.wikimedia.org/r/375352TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
gerritbot added a comment.
Change 375741 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] mediawiki: make the wikidata wb_terms rebuild a little bit faster
https://gerrit.wikimedia.org/r/375741TASK
gerritbot added a comment.
Change 375352 had a related patch set uploaded (by Thiemo Mättig (WMDE); owner: Thiemo Mättig (WMDE)):
[mediawiki/extensions/Wikibase@master] Fix minor issues recently introduced in TermSqlIndexBuilder and test
https://gerrit.wikimedia.org/r/375352TASK
gerritbot added a comment.
Change 374342 merged by Volans:
[operations/puppet@production] mediawiki: fix logrotating in wikidata cronjob (2)
https://gerrit.wikimedia.org/r/374342TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
gerritbot added a comment.
Change 374342 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] mediawiki: fix logrotating in wikidata cronjob (2)
https://gerrit.wikimedia.org/r/374342TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
gerritbot added a comment.
Change 373854 merged by Volans:
[operations/puppet@production] mediawiki: fix logrotating in wikidata cronjob
https://gerrit.wikimedia.org/r/373854TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
gerritbot added a comment.
Change 373854 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] mediawiki: fix logrotating in wikidata cronjob
https://gerrit.wikimedia.org/r/373854TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
jcrespo added a comment.
@Ladsgroup: This should be easy to fix:
root@terbium:/var/log/wikidata$ ls -lha rebuildTermSql*
-rw-rw-r-- 1 www-data www-data 1.7K Aug 25 06:30 rebuildTermSqlIndex.log
-rw-r--r-- 1 www-data www-data 130 Aug 8 13:15 rebuildTermSqlIndex.log-20170810.gz
-rw-rw-r-- 1
gerritbot added a comment.
Change 373507 merged by Jcrespo:
[operations/puppet@production] wikidata-maintenance: Emergency stop of rebuildTermSqlIndex
https://gerrit.wikimedia.org/r/373507TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
jcrespo added a comment.
So this is deployed into production, we did a test run and it seems to work as intended.
I left a "disable" patch https://gerrit.wikimedia.org/r/373507 and instructions to deploy there, in case fellow ops have to do some emergency thing to disable, so it is already ready
gerritbot added a comment.
Change 373507 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] wikidata-maintenance: Emergency stop of rebuildTermSqlIndex
https://gerrit.wikimedia.org/r/373507TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
gerritbot added a comment.
Change 370626 merged by Jcrespo:
[operations/puppet@production] mediawiki: Add puppetized cronjob for rebuildTermSqlIndex
https://gerrit.wikimedia.org/r/370626TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-08-24T07:24:56Z] starting the run for rebuildTermIndex (T171460)TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup,
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-08-22T10:21:38Z] another run of rebuildTermSqlIndex (T171460)TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, StashbotCc:
gerritbot added a comment.
Change 372533 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add sleep option to the rebuildTermSqlIndex maintenance script
https://gerrit.wikimedia.org/r/372533TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
Ladsgroup added a comment.
Now it has been done up to Q200,000
Processed up to page 191988 (Q194046)
Processed up to page 192997 (Q195126)
Processed up to page 194002 (Q196205)
Processed up to page 195008 (Q197508)
Processed up to page 196029 (Q198703)
Processed up to page 197056 (Q200160)
I
gerritbot added a comment.
Change 372533 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Add sleep option to the rebuildTermSqlIndex maintenance script
https://gerrit.wikimedia.org/r/372533TASK
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-08-18T09:55:33Z] one small pass of ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=entity (T171460)TASK
Ladsgroup added a comment.
All properties have labels and:
ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property
Processed up to page 18348389 (P1289)
Processed up to page
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-08-18T09:46:57Z] ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property (T171460)TASK
Ladsgroup added a comment.
I confirm after deploying wmf.14 the maintainance script becomes way faster and also won't remove any labels from testwikidata. i.e. we can run the script in prod starting tomorrow.TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-08-16T12:25:53Z] ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki testwikidatawiki --entity-type=property (T172776, T171460)TASK
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-08-08T11:22:12Z] start of ladsgroup@terbium:~$ timeout 3500s /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item
gerritbot added a comment.
Change 370626 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[operations/puppet@production] mediawiki: Add puppetized cronjob for rebuildTermSqlIndex
https://gerrit.wikimedia.org/r/370626TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
Ladsgroup added a comment.
Since there 854M rows in wb_terms right now, my estimation is that it will take 64 days.TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: LadsgroupCc: Stashbot, Marostegui, aude,
Ladsgroup added a comment.
Properties are done now, since the number was small, I thought let's run it with "--deduplicate-terms" flag but it caused the terms in Wikidata to disappear temporarily (as was noted in #wikidata irc channel). So I stopped and ran it without that flag.
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-08-08T07:34:17Z] stopped the script and re-running without --deduplicate-terms (T171460)TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-08-08T07:21:33Z] start of ladsgroup@terbium:~$ time mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki --entity-type=property --deduplicate-terms (T171460)TASK
Marostegui added a comment.
Awesome!! :-)
Thank youTASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Marostegui, aude, Ladsgroup, Aklapper, hoo, PokestarFan, daniel, GoranSMilovanovic, QZanden,
daniel added a comment.
In T171460#3465634, @Marostegui wrote:
I assume this script has the proper throttling measures: ie, wait for replication?
It does - if anything, it waits for replication more often than necessary.TASK DETAILhttps://phabricator.wikimedia.org/T171460EMAIL
34 matches
Mail list logo
