[Wikidata-bugs] [Maniphest] [Commented On] T177550: Only dump up to N entities in each maintenance script run
gerritbot added a comment. Change 405739 merged by jenkins-bot: [mediawiki/extensions/Wikibase@master] Allow continuing Wikibase entity dumps https://gerrit.wikimedia.org/r/405739TASK DETAILhttps://phabricator.wikimedia.org/T177550EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: Lucas_Werkmeister_WMDE, gerritbot, Liuxinyu970226, Lydia_Pintscher, daniel, ArielGlenn, Aklapper, hoo, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Wikidata-bugs, aude, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T177550: Only dump up to N entities in each maintenance script run
gerritbot added a comment. Change 405739 had a related patch set uploaded (by Hoo man; owner: Hoo man): [mediawiki/extensions/Wikibase@master] Allow continuing Wikibase entity dumps https://gerrit.wikimedia.org/r/405739TASK DETAILhttps://phabricator.wikimedia.org/T177550EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Liuxinyu970226, Lydia_Pintscher, daniel, ArielGlenn, Aklapper, hoo, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T177550: Only dump up to N entities in each maintenance script run
hoo added a comment. In T177550#3904087, @ArielGlenn wrote: Can you pass the script a first and last entity id and get around the problem that way? I looked into this, but I don't think it is viable: We use the page id to paging internally, and a higher entity id doesn't imply a higher page id (most of the time, but not always). Due to this I chose to implement this using offsets.TASK DETAILhttps://phabricator.wikimedia.org/T177550EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: Liuxinyu970226, Lydia_Pintscher, daniel, ArielGlenn, Aklapper, hoo, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T177550: Only dump up to N entities in each maintenance script run
ArielGlenn added a comment. Can you pass the script a first and last entity id and get around the problem that way?TASK DETAILhttps://phabricator.wikimedia.org/T177550EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArielGlennCc: Liuxinyu970226, Lydia_Pintscher, daniel, ArielGlenn, Aklapper, hoo, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T177550: Only dump up to N entities in each maintenance script run
hoo added a comment. In T177550#3901012, @ArielGlenn wrote: How is this coming along? No work has been done on this yet, but it shouldn't be to hard. At very least we will need a --continue parameter for the dump maintenance script. Also we will need to find a way to stop… keep dumping in batches of say 100,000 until the batches are empty? How would we know from where to continue? Grep this from the logs or do we want a special "state file" for this that has the entity id last dumped by that shard?TASK DETAILhttps://phabricator.wikimedia.org/T177550EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hooCc: Liuxinyu970226, Lydia_Pintscher, daniel, ArielGlenn, Aklapper, hoo, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T177550: Only dump up to N entities in each maintenance script run
ArielGlenn added a comment. How is this coming along?TASK DETAILhttps://phabricator.wikimedia.org/T177550EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ArielGlennCc: Liuxinyu970226, Lydia_Pintscher, daniel, ArielGlenn, Aklapper, hoo, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Wikidata-bugs, aude, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
