Haudy Kazemi
Wed, 01 Jul 2009 05:12:34 -0700
Ian Collins wrote:
The way Google handles UPSes is to have a small 12v battery integrated with each PC power supply. When the machine is on, the battery has its charged maintained. Not unlike a laptop in that it has a built in battery backup, but using an inexpensive sealed lead acid battery instead of lithium ion. Here is info along with photos of the Google server internals:David Magda wrote:Or run your systems of DC and get as much backup as you have room (and budget!) for batteries. I once visited a central exchange with 48 hours of battery capacity...On Jun 30, 2009, at 14:08, Bob Friesenhahn wrote:I have seen UPSs help quite a lot for short glitches lasting seconds, or a minute. Otherwise the outage is usually longer than the UPSs can stay up since the problem required human attention.A standby generator is needed for any long outages.Can't remember where I read the claim, but supposedly if power isn't restored within about ten minutes, then it will probably be out for a few hours. If this 'statistic' is true, it would mean that your UPS should last (say) fifteen minutes, and after that you really need a generator.
http://news.cnet.com/8301-1001_3-10209580-92.html http://willysr.blogspot.com/2009/04/googles-server-design.html(IIRC there have been power supply UPSes since at least the late 1980s which had an internal battery. Either that or they were UPSes that fit inside the standard PC (AT) compatible desktop case, making the power protection system entirely internal to the computer. I think I saw these models one time while browsing late 1980s or early 1990s issues of PC Magazine that reviewed UPSes. They still exist...one company selling them is http://www.globtek.com/html/ups.html . A Google search for 'power supply built in UPS' would likely find more.)
I also did additional searches in the zfs-discuss archives and found a thread from mid-February, which lead me to other threads. It looks like there are still scattered instances where ZFS has not recovered gracefully from power failures or other failures, where it became necessary to perform a manual transaction group (txg) rollback. Here is a consolidated list of links related to manual uberblock transaction group (txg) rollback and similar ZFS data recovery guides, including undeleting:
Section 1: Nathan Hand's guide and related thread Nathan Hand's guide to invalidating uberblocks (Dec 2008 thread) http://www.opensolaris.org/jive/thread.jspa?threadID=85794 or http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg22153.html Section 2. Victor Latushkin's guide and related threadsThread: zpool unimportable (corrupt zpool metadata??) but no zdb -l device problems (Oct 2008 to Feb 2009 thread)
http://www.opensolaris.org/jive/thread.jspa?threadID=76960 or http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg19839.html Repair report: Re: Solved - a big THANKS to Victor Latushkin @ Sun / Moscow http://www.opensolaris.org/jive/message.jspa?messageID=289537#289537Some recovery discussion by Victor: "zdb -bv alone took several hours to walk the block tree"
http://www.opensolaris.org/jive/message.jspa?messageID=292991#292991or http://mail.opensolaris.org/pipermail/zfs-discuss/2008-October/022365.html
or http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg20095.htmlVictor Latushkin's guide: "Thanks to COW nature of ZFS it was possible to successfully recover pool state which was only 5 seconds older than last unopenable one."
http://mail.opensolaris.org/pipermail/zfs-discuss/2008-October/022331.html or http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg20061.html Section 3: reliability debates, recovery tool planning, uberblock infoThread: Availability: ZFS needs to handle disk removal / driver failure better (August 2008 thread)
http://www.opensolaris.org/jive/thread.jspa?threadID=70811 or http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg19057.html Thread: ZFS: unreliable for professional usage? (Feb 2009 thread) http://www.opensolaris.org/jive/thread.jspa?threadID=91426 or http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg23833.htmlRichard Elling's post that "uberblocks are kept in an 128-entry circular queue which is 4x redundant with 2 copies each at the beginning and end of the vdev. Other metadata, by default, is 2x redundant and spatially diverse."
http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg24145.html Jeff Bonwick's post about Bug ID 6667683 http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg23961.html Bug ID 6667683: need a way to rollback to an uberblock from a previous txgDescription: If we are unable to open the pool based on the most recent uberblock then it might be useful to try an older txg uberblock as it might provide a better view of the world. Having a utility to reset the uberblock to a previous txg might provide a nice recovery mechanism.
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6667683 Uberblock information http://blogs.sun.com/blogfinger/entry/zfs_and_the_uberblock http://blogs.sun.com/blogfinger/entry/zfs_and_the_uberblock_part Section 4: undeletingRecovering removed file on zfs disk using a modified mdb and zdb (i.e. undelete)
http://mbruning.blogspot.com/2008/08/recovering-removed-file-on-zfs-disk.htmlRe: [zfs-discuss] Forensic analysis [was: more ZFS recovery] (listed because forensic analysis tools often overlap with undeletion tools/data recovery tools)
http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg18557.html http://opensolaris.org/os/project/forensics/ZFS-Forensics/ Thanks everyone for the input you've given so far. -hk _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss