forwarding this, it seems to not have made it to the labs list last
time and I didn't see it waiting for moderation.
Ariel
--- Begin Message ---
Probably no one else does know (how to run these dumps).
So to clear up a little of the confusion/lack of information,
automating staged dumps is currently tracked in Phabricator here:
https://phabricator.wikimedia.org/T107750
Just got the configuration files puppetized for the scheduler, which
was started manually this time but won't need any other intervention.
I'll be keeping an eye on it anyways since this too is new code.
I've been playing whack-a-mole with bugs in the streaming stubs/logs
production, which is why the delay again last month in completing enwp
and why only one set of dumps went off. This run should be better,
cross fingers that nothing else crept in.
After one complete run finishes for the non en wikipedia dumps, we'll
be able to do a second run, probably leaving off revision history since
most folks don't need those more than once a month. <-- If this is
wrong, speak up now.
The order in which things are run has been redone a bit, hopefully to
get the parts of the dumps people need done sooner in the cycle.
I'm still reluctant to do partial dump copies over labs unless we can
guarantee a little more space over there to keep e.g. the last full
good dump along with whatever new files are being copied. I've opened
a Phabricator task for that here
https://phabricator.wikimedia.org/T108077 and added Coren so we can
figure out the space issue.
I've not wanted to weigh in earlier until I had things in place to kick
off the new run. My apologies if that just made folks more annoyed.
Ariel
Στις 04-08-2015, ημέρα Τρι, και ώρα 20:08 -0600, ο/η Bryan White
έγραψε:
> Again, I was on the message you mentioned. I was the person Ariel
> was
> responding too. I have been in touch with Ariel in the past, but he
> not responded to any emails from me or others for almost three weeks
> about the situation. Problems in Greece, on holiday, real-life,
> ...?
>
> > > > Last month, French, German, Japanese, Russian, Spanish and
> > > > others were never completed.
>
> I use over 40 languages, not just enwiki. The one's that didn't
> finish are useless for the majority of people. It takes two weeks to
> dump all languages except for enwiki and only one has started in 18
> days. By "started", I mean re-started a pre-existing dump to
> completion. No new dump has commenced in over a month. For many
> languages (French, German, etc) it has been two months. Things are
> dead.
>
> Does anybody know how to get dumps working who's name isn't Ariel?
>
> Bryan
>
>
> On Tue, Aug 4, 2015 at 7:36 PM, Hydriz Scholz <[email protected]>
> wrote:
> > Hello,
> >
> > I apologize if I was not clear enough. The dumps are being run
> > manually because Ariel is currently testing the new framework. It
> > is thus to be expected if the dumps are not being produced
> > according to schedule. Also, I believe the enwiki dump is still
> > being produced, so I doubt that the process is "dead", it's just
> > slow.
> >
> > Perhaps Ariel is able to provide more information on this issue, as
> > I am unable to comment further than this.
> >
> > > On 5 Aug 2015, at 08:38, Bryan White <[email protected]> wrote:
> > >
> > > I did mention that dumps are now run manually. I was on the
> > > message
> > > you mentioned. Only one dump has started in the last 18 days,
> > > this
> > > is in addition to larger languages never completing. Going
> > > from two
> > > dumps a month to no dumps a month is not a "delay", it is dead.
> > >
> > > Bryan
> > >
> > > > On Tue, Aug 4, 2015 at 6:08 PM, Hydriz Scholz <
> > > > [email protected]> wrote:
> > > > Hello,
> > > >
> > > > Ariel noted last month that the dumps are currently being ran
> > > > manually. [1] Having delays in the dumps production should thus
> > > > be expected.
> > > >
> > > > Hope this helps.
> > > >
> > > > [1]: https://lists.wikimedia.org/pipermail/xmldatadumps-l/2015-
> > > > July/001153.html
> > > >
> > > > > On 5 Aug 2015, at 07:26, Bryan White <[email protected]>
> > > > > wrote:
> > > > >
> > > > > Three months ago, "they" changed the way dumps are done.
> > > > > "They"
> > > > > wanted stubs for every language done first and work on the
> > > > > rest of the
> > > > > dump later.
> > > > >
> > > > > Old way: Two dumps a month for all languages except enwiki.
> > > > > A new
> > > > > dump was started when one completed. Except for larger
> > > > > languages,
> > > > > dumps were done in under a day.
> > > > >
> > > > > New way: One dump a month for majority of languages. Dumps
> > > > > have to
> > > > > be started manually. All dumps take 2-4 weeks to complete.
> > > > >
> > > > > Last month, French, German, Japanese, Russian, Spanish and
> > > > > others were
> > > > > never completed.
> > > > >
> > > > > It's about to be August 5. No dumps have started this month.
> > > > > Only
> > > > > one dump has started in over two weeks.
> > > > >
> > > > >
> > > > > Does anybody know how to get dumps working again.
> > > > >
> > > > > Bryan
> > > > >
> > > > > _______________________________________________
> > > > > Labs-l mailing list
> > > > > [email protected]
> > > > > https://lists.wikimedia.org/mailman/listinfo/labs-l
> > > >
> > > > _______________________________________________
> > > > Labs-l mailing list
> > > > [email protected]
> > > > https://lists.wikimedia.org/mailman/listinfo/labs-l
> > >
> > > _______________________________________________
> > > Labs-l mailing list
> > > [email protected]
> > > https://lists.wikimedia.org/mailman/listinfo/labs-l
> >
> > _______________________________________________
> > Labs-l mailing list
> > [email protected]
> > https://lists.wikimedia.org/mailman/listinfo/labs-l
>
> _______________________________________________
> Xmldatadumps-l mailing list
> [email protected]
> https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l
--- End Message ---
_______________________________________________
Labs-l mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/labs-l
