[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-12-06 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-12-06T20:15:26Z] Ran "scap pull" on snapshot1001 after T177486 related testsTASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo,

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-11-21 Thread gerritbot
gerritbot added a comment. Change 392670 merged by ArielGlenn: [operations/puppet@production] Set Wikidata entity dump batch size to 1500 https://gerrit.wikimedia.org/r/392670TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-11-21 Thread gerritbot
gerritbot added a comment. Change 392670 had a related patch set uploaded (by Hoo man; owner: Hoo man): [operations/puppet@production] Set Wikidata entity dump batch size to 1500 https://gerrit.wikimedia.org/r/392670TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-11-13 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-11-13T18:09:05Z] Ran "scap pull" on mwdebug1001/snapshot1001 after (further) tests re T177486TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-11-13 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-11-13T17:28:13Z] Ran "scap pull" on mwdebug1001 after tests re T177486TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo,

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-11-01 Thread daniel
daniel added a comment. @hoo we should really look into generating RDF from JSON. Can probably be done in a week. That would mean moving a lot less data from storage over the network. Should be faster. How much, I can't say...TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-10-31 Thread ArielGlenn
ArielGlenn added a comment. In T177486#3724106, @hoo wrote: (Probably) due to the DataModel updates the current JSON dump was created in just 25 hours, compared to ~34-35h last week. (This is data from one run only, so not overly reliable… but the difference is huge) If all future runs turn out

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-10-31 Thread hoo
hoo added a comment. (Probably) due to the DataModel updates the current JSON dump was created in just 25 hours, compared to ~34-35h last week. (This is data from one run only, so not overly reliable… but the difference is huge)TASK DETAILhttps://phabricator.wikimedia.org/T177486EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T177486: [Tracking] Wikidata entity dumpers need to cope with the immense Wikidata growth recently

2017-10-30 Thread hoo
hoo added a comment. While T178247: Use a retrieve only CachingEntityRevisionLookup for dumps will certainly make the dumps much faster, it will only do so (noticeably) on HHVM. This is because we split the cache between HHVM and Zend (see below), thus the (currently) Zend dumpers wont profit from