FYI ---------- Forwarded message ---------- From: Ariel Glenn WMF <[email protected]> Date: Mon, Sep 12, 2016 at 9:07 AM Subject: [Research-Internal] Fwd: Dumps Rewrite getting underway (help needed!) To: [email protected]
---------- Forwarded message ---------- From: Ariel Glenn WMF <[email protected]> Date: Mon, Sep 5, 2016 at 2:35 PM Subject: Dumps Rewrite getting underway (help needed!) To: Wikipedia Xmldatadumps-l <[email protected]> Hello folks, I know a number of you have subscribed to the Dumps Rewrite project ( https://phabricator.wikimedia.org/tag/dumps-rewrite/) but I bet none of you actually watch it or any of its tasks. So here's a heads up. I'm getting started on work on the job scheduler/workflow manager piece; this would accept lists of dump tasks (in the current setup, "dump stubs for el wikipedia"), call a callback to turn each of them into small jobs that can be completed in less than an hour, submit and monitor these jobs with retries, dependencies etc, call a callback to recombine the outputs of the jobs, and notify some caller on success of te whole operation. First up is evaluating existing packages and choosing one to use as a foundation. Please contribute! See the following tasks: https://phabricator.wikimedia.org/T143205: Draft usage scenarios for job/workflow manager <https://phabricator.wikimedia.org/T143205> https://phabricator.wikimedia.org/T143206: List requirements needed for task/job/workflow manager <https://phabricator.wikimedia.org/T143206> https://phabricator.wikimedia.org/T143207: Evaluate software packages for job/task/workflow management <https://phabricator.wikimedia.org/T143207> Also, can someone please forward this on to analytics-l and research-l? I'm not on those lists but they will no doubt have a lot of useful expertise here. Thanks! Ariel _______________________________________________ Research-Internal mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/research-internal
_______________________________________________ Analytics mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/analytics
