[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-12-06 Thread Stashbot 
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-12-06T20:15:26Z]  Ran "scap pull" on snapshot1001 after T177486 related testsTASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, StashbotCc: Stashbot, Sjoerddebruin, gerritbot, thiemowmde, Aklapper, ezachte, daniel, Lydia_Pintscher, mark, ArielGlenn, bd808, Liuxinyu970226, aude, JanZerebecki, Jimkont, Denis.bykov, Ricordisamoa, PokestarFan, hoo, Lahi, Gq86, GoranSMilovanovic, QZanden, Wikidata-bugs, Svick, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-11-21 Thread gerritbot 
gerritbot added a comment.
Change 392670 merged by ArielGlenn:
[operations/puppet@production] Set Wikidata entity dump batch size to 1500

https://gerrit.wikimedia.org/r/392670TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: Stashbot, Sjoerddebruin, gerritbot, thiemowmde, Aklapper, ezachte, daniel, Lydia_Pintscher, mark, ArielGlenn, bd808, Liuxinyu970226, aude, JanZerebecki, Jimkont, Denis.bykov, Ricordisamoa, PokestarFan, hoo, Lahi, Gq86, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Wikidata-bugs, Svick, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-11-21 Thread gerritbot 
gerritbot added a comment.
Change 392670 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[operations/puppet@production] Set Wikidata entity dump batch size to 1500

https://gerrit.wikimedia.org/r/392670TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, gerritbotCc: Stashbot, Sjoerddebruin, gerritbot, thiemowmde, Aklapper, ezachte, daniel, Lydia_Pintscher, mark, ArielGlenn, bd808, Liuxinyu970226, aude, JanZerebecki, Jimkont, Denis.bykov, Ricordisamoa, PokestarFan, hoo, Lahi, Gq86, GoranSMilovanovic, QZanden, Wikidata-bugs, Svick, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-11-13 Thread Stashbot 
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-11-13T18:09:05Z]  Ran "scap pull" on mwdebug1001/snapshot1001 after (further) tests re T177486TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, StashbotCc: Stashbot, Sjoerddebruin, gerritbot, thiemowmde, Aklapper, ezachte, daniel, Lydia_Pintscher, mark, ArielGlenn, bd808, Liuxinyu970226, aude, JanZerebecki, Jimkont, Denis.bykov, Ricordisamoa, PokestarFan, hoo, Lahi, GoranSMilovanovic, QZanden, Wikidata-bugs, Svick, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-11-13 Thread Stashbot 
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-11-13T17:28:13Z]  Ran "scap pull" on mwdebug1001 after tests re T177486TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, StashbotCc: Stashbot, Sjoerddebruin, gerritbot, thiemowmde, Aklapper, ezachte, daniel, Lydia_Pintscher, mark, ArielGlenn, bd808, Liuxinyu970226, aude, JanZerebecki, Jimkont, Denis.bykov, Ricordisamoa, PokestarFan, hoo, Lahi, GoranSMilovanovic, QZanden, Wikidata-bugs, Svick, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-11-01 Thread daniel 
daniel added a comment.
@hoo we should really look into generating RDF from JSON. Can probably be done in a week.

That would mean moving a lot less data from storage over the network. Should be faster. How much, I can't say...TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, danielCc: Sjoerddebruin, gerritbot, thiemowmde, Aklapper, ezachte, daniel, Lydia_Pintscher, mark, ArielGlenn, bd808, Liuxinyu970226, aude, JanZerebecki, Jimkont, Denis.bykov, Ricordisamoa, PokestarFan, hoo, Lahi, GoranSMilovanovic, QZanden, Wikidata-bugs, Svick, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-10-31 Thread ArielGlenn 
ArielGlenn added a comment.

In T177486#3724106, @hoo wrote:
(Probably) due to the DataModel updates the current JSON dump was created in just 25 hours, compared to ~34-35h last week. (This is data from one run only, so not overly reliable… but the difference is huge)


If all future runs turn out that way, this is very good news! Looking forward to the other optimizations too.TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, ArielGlennCc: Sjoerddebruin, gerritbot, thiemowmde, Aklapper, ezachte, daniel, Lydia_Pintscher, mark, ArielGlenn, bd808, Liuxinyu970226, aude, JanZerebecki, Jimkont, Denis.bykov, Ricordisamoa, PokestarFan, hoo, Lahi, GoranSMilovanovic, QZanden, Wikidata-bugs, Svick, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-10-31 Thread hoo 
hoo added a comment.
(Probably) due to the DataModel updates the current JSON dump was created in just 25 hours, compared to ~34-35h last week. (This is data from one run only, so not overly reliable… but the difference is huge)TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: Sjoerddebruin, gerritbot, thiemowmde, Aklapper, ezachte, daniel, Lydia_Pintscher, mark, ArielGlenn, bd808, Liuxinyu970226, aude, JanZerebecki, Jimkont, Denis.bykov, Ricordisamoa, PokestarFan, hoo, Lahi, GoranSMilovanovic, QZanden, Wikidata-bugs, Svick, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-10-30 Thread hoo 
hoo added a comment.
While T178247: Use a retrieve only CachingEntityRevisionLookup for dumps will certainly make the dumps much faster, it will only do so (noticeably) on HHVM. This is because we split the cache between HHVM and Zend (see below), thus the (currently) Zend dumpers wont profit from the cache which is probably mostly populated in the HHVM version of the cache (as all app server run HHVM).
There are some other maintenance scripts using Zend which might also write into this cache… so maybe this will still help something, though.

if ( defined( 'HHVM_VERSION' ) ) {
// Split the cache up for hhvm. T73461
$wgWBSharedCacheKey .= '-hhvm';
}TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: Sjoerddebruin, gerritbot, thiemowmde, Aklapper, ezachte, daniel, Lydia_Pintscher, mark, ArielGlenn, bd808, Liuxinyu970226, aude, JanZerebecki, Jimkont, Denis.bykov, Ricordisamoa, PokestarFan, hoo, Lahi, GoranSMilovanovic, QZanden, Wikidata-bugs, Svick, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs