> Date: Wed, 17 Jun 2015 23:09:11 -0400
> From:
[email protected]
> To:
[email protected];
[email protected]
> Subject: Re: [Labs-l] NFS outage in progress [UPDATE]
>
> On 15-06-17 09:29 PM, Andrew Bogott wrote:
> > Coren is rebooting and fscking the system -- with luck it'll be up
> > again within the hour.
>
> Fortune does not smile upon us; the fsck is taking a LONG time to
> progress (over a 40T filesystem), but the repair goes apace.
>
> There is already indication of some lost data - though the failures are
> likely to date from a past crash and may have gone unnoticed for some time.
>
> Either way, we have a fairly recent (8-9 days) backup of the entire
> filesystem so we may be able to recover lost or corrupted files if any
> turn up.
>
> Worst case scenario is a catastrophic loss of the entire filesystem
> (which does not seem likely at this time), in which case we will restore
> the filesystem completely from the backup.
>
> The repair process will take at least a few more hours, during which I
> will be keeping watch. I will update this list as soon as I have a
> better assessment of the damage, if any.
>
> -- Marc
>
>
> _______________________________________________
> Labs-l mailing list
>
[email protected]
>
https://lists.wikimedia.org/mailman/listinfo/labs-l