[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. Note that if you don't get the emails for the tasks you might not be watching this project, only a member. So check that too. TASK DETAIL https://phabricator.wikimedia.org/T114019 EMAIL PREFERENCES

Re: [Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

no On Fri, Dec 11, 2015 at 5:42 AM, ArielGlenn < no-re...@phabricator.wikimedia.org> wrote: > ArielGlenn added a comment. > > In a conversation on IRC robla suggested that this would be a very useful > question so I post it here: > > "do we need a new multi-format dump architecture to replace

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. Yes indeed, Flow and anything folks want to produce in the future. We want this to be as easy as adding a config section to a puppet manifest (once the script to produce the dataset is written and tested). TASK DETAIL https://phabricator.wikimedia.org/T114019

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

Mattflaschen added a comment. > Content. We have WikiData which has its own unique content; should we do something special here? Flow now has dumps, though they're not running in production yet (open task). Flow would need to fit into this framework, though. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. Draft questions now at https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit_2016/T114019/Minutes/Questions Please edit away. If you edit there please comment here so that task watchers (like me) know to go check the page for new stuff. Thanks! TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. Yes indeed, I was going to start the draft list here and ask people to please chime in/add/remove/fix. TASK DETAIL https://phabricator.wikimedia.org/T114019 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ArielGlenn Cc:

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

Milimetric added a comment. nice job organizing, Ariel, let me know if you want to bounce the questions off someone TASK DETAIL https://phabricator.wikimedia.org/T114019 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ArielGlenn, Milimetric Cc:

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. AWight's note are now available on the session notes; Halfak's notes are also available there. Next up: cull questions that need to be answered in order to work on implementation. TASK DETAIL https://phabricator.wikimedia.org/T114019 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. AWight feedback to be gathered async, he's got a very full plate. So it might be put off til later, will find out soon. TASK DETAIL https://phabricator.wikimedia.org/T114019 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. Will chat with @Halfak tomorrow at around 18.00 my time i.e. EET (16.00 UTC I guess). Notes to go to etherpad first and then wiki page as usual. After that AWight and then done with info gathering for this round. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. https://etherpad.wikimedia.org/p/WikiDev16-T114019 has rought notes from talk with millimetric's team, I will clean these up and move to the wiki page for the dev session soon. AWight and maybe one more Aaron are next. Then it will be question culling time. TASK

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

Milimetric added a comment. @ArielGlenn - looking forward to the discussion TASK DETAIL https://phabricator.wikimedia.org/T114019 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ArielGlenn, Milimetric Cc: jberkel, NealMcB, jcrespo, Bianjiang,

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. https://phabricator.wikimedia.org/tag/dumps-rewrite/ New project has been created, I've added you all to it I think, but please check to be sure. On this task I need to chat with Milimetric and AWight and record those notes, then cull out questions for followup.

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. The session notes have been updated with notes from a later discussion with Gwicke: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit_2016/T114019/Minutes#Gabriel_Wicke_discussion_notes Next up: AWight TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

Aklapper added a comment. Wikimedia Developer Summit 2016 ended two weeks ago. This task is still open. **If the session in this task took place**, please make sure 1) that the session Etherpad notes are linked from this task, 2) that followup tasks for any actions identified have been created

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. notes are linked, also copied to wiki page so etherpad can go away, followup task generation is in progress. Once that is complete this task will be resolved. TASK DETAIL https://phabricator.wikimedia.org/T114019 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. We have session notes here: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit_2016/T114019 Lots to process still. I'm going to chat with Adam Wight (didn't get to do that at the dev summit) and also Gabriel Wicke (same) this week and add those notes to the

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. I have scheduled this session on the Unconference track for tomorrow, Jan 5 at 10 a.m. We'll be identifying the main issues for users of the dumps (I already know what the main issues are for the maintainer!), and then discussing approaches to address those issues

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

GWicke added a comment. @ArielGlenn: To me it seems that the discussion so far lacks a shared agreement on what the most pressing problems with dumps are. This makes it difficult to evaluate candidate solutions and their trade-offs relative to the top priorities. With the right preparation, a

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. Problem 1 does not need to wait for dumps 2.0; the instant workaround (though a workaround and not a solution) is to download from the your.org mirror which provides reasonable download speeds. In the meantime I will open a ticket for this issue and cc you on it,

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. OK, I'll backtrack then. There's some problems listed in the description but maybe none of those is "the most important problem to solve" with the current system. So I"m taking bids, what do people think? And yes as the maintainer I have an opinion but it's as a

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. In a conversation on IRC robla suggested that this would be a very useful question so I post it here: "do we need a new multi-format dump architecture to replace our XML-based system, or is there a better approach?" TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

RobLa-WMF added a comment. I think there's a risk of second system syndrome here in the current proposal. I would start with the question "what's the most important problem to solve with the current system?" My

[Wikidata-bugs] [Maniphest] [Commented On] T114019: Dumps 2.0 for realz (planning/architecture session)

ArielGlenn added a comment. First, the same old Dumps 1.0 are what are running now. If we don't get some collaboration on 2.0 there won't be a 2.0. And that's exactly what I meant when I said that if people don't show up we'll just have the same old dumps we always had. Second, for current