You may have noticed that archiving was suspended at The Mail Archive recently. Things are fine now, read on if you want gory details.
We are hosted at a professional datacenter, complete with building wide uninterruptible power supply (UPS) and backup generators. About 5 years ago, the datacenter botched some maintenance work and accidentally cut power. In response, we deployed an individual UPS on our primary server to make things even safer. That turned out to be a mistake. On Thursday that UPS failed in the worst possible way, abruptly dropping power while a lot of data was in flight. This is normally just an annoyance, but in this case it caused enough damage to the filesystem that we had suspend archiving and switch to read-only mode. To fix things right, we express ordered and installed some additional storage. Because everything is now on large solid state drives (SSD) we can afford switching to a filesystem that is tuned more towards robustness, at the cost of some performance. For the filesystem junkies that means migrating from the fairly exotic XFS setup below to a fairly stock EXT4 setup. The new filesystem is about 20% less space efficient, but that's okay. We now have enough room for years to come. It took almost a day to fully diagnose, an overnight parts delivery, a few hours to get everything set up correctly, then 10 more hours to move all the data. We did not have to resort to restoring from backups, but they are certainly there if we need it. I'd like to emphasize that the data is safe. We were able to reconstruct everything that was in flight at the time of the outage. And while we had archiving paused, inbound mail was queuing up patiently. The system is crazy fast and we burned off the archiving backlog in just a few hours. Thanks for your patience and I hope you enjoyed this peek into what goes on behind the scenes. I think the biggest benefit of using a service like The Mail Archive is we get the fun of dealing with problems like this so you don't have to. Cheers, Jeff === mkfs.xfs -n size=16k -i attr=2 -l lazy-count=1,version=2,size=32m \ -b size=512 noatime,logbufs=8,logbsize=256k mount -onoatime,logbufs=8,logbsize=256k
_______________________________________________ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com http://mail-archive.com/cgi-bin/mailman/options/gossip
