Wow. I'm really really sorry about this. I'm guessing it's my fault based on the timing. In order to make up for this, I'll try and contact enough users to re-assemble the logs for the missing days. My bouncer should have a good portion of the logs to start with. Petr, can you give me a list of the channels that are missing and the timestamps to start and end from? -- Legoktm http://enwp.org/User:Legoktm
On Sat, Dec 15, 2012 at 7:12 AM, Petr Bena <[email protected]> wrote: > I found out exactly what happened, though I don't know who did it. The > whole logs folder was recursively deleted 4 days ago, interesting on this > is that the bot is only creating the log folders when the channel config is > loaded and log folder doesn't exist. Because the bot was running and all > channels were loaded, it was attempting to flush its own cache to disk but > it wasn't able to do so because the path of target file wasn't valid (when > you write to a file, it doesn't automatically create the folders). That's > why we are missing logs for so many days - the bot simply wasn't producing > any, backup is running every day but there was nothing to backup, and > yesterday when I inserted new module for labs tools, the bot needed to > reboot, and that was the time when folders were created and logging started > to continue. > > > > On Sat, Dec 15, 2012 at 1:45 AM, Petr Bena <[email protected]> wrote: > >> of course that's not the way I prefer we need to find out what is >> happening, it may be even some my mistake or some weird bug, there is a >> script in cron which generates the .gz files, that one is running as root >> as far as I know, because it creates the backups too and sets some >> permissions on them, it's cronned on bots-1 but I didn't change it for >> weeks and it never happened, so i don't think it has something to do with >> that... >> >> >> >> On Sat, Dec 15, 2012 at 1:38 AM, Jeremy Baron <[email protected]>wrote: >> >>> On Fri, Dec 14, 2012 at 7:12 PM, Jeremy Baron <[email protected]> >>> wrote: >>> > On Fri, Dec 14, 2012 at 6:55 PM, Petr Bena <[email protected]> wrote: >>> >> Hi, I don't know what happened, but someone removed recursively all >>> data in >>> >> ~wm-bot/logs >>> > >>> > Wasn't that accident already announced? >>> >>> erm, whooops. seems I misinterpreted >>> http://lists.wikimedia.org/pipermail/labs-l/2012-December/000553.html >>> and that prior deletion was narrower than you've just noticed? >>> >>> >> I may have a backup but because this removal was done yesterday or >>> earlier >>> >> and apparently happens every day, it's a problem, because even the >>> complete >>> >> logs archive is almost empty now. >>> > >>> > Every day?? elaborate? >>> >>> Well, I'm still not too clear on this point but if this really is >>> happening repeatedly (cron'd??) then we need to track down what's >>> doing it. We can't be needing to manually restore backups multiple >>> times per week. (and we'd probably have incremental data loss between >>> last backup and deletion too) >>> >>> -Jeremy >>> >>> _______________________________________________ >>> Labs-l mailing list >>> [email protected] >>> https://lists.wikimedia.org/mailman/listinfo/labs-l >>> >> >> > > _______________________________________________ > Labs-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/labs-l > >
_______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
