Re: [zfs-discuss] Recovering lost labels on raidz member
On Aug 13, 2012, at 8:59 PM, Scott wrote: > On Mon, Aug 13, 2012 at 10:40:45AM -0700, Richard Elling wrote: >> >> On Aug 13, 2012, at 2:24 AM, Sa?o Kiselkov wrote: >> >>> On 08/13/2012 10:45 AM, Scott wrote: Hi Saso, thanks for your reply. If all disks are the same, is the root pointer the same? >>> >>> No. >>> Also, is there a "signature" or something unique to the root block that I can search for on the disk? I'm going through the On-disk specification at the moment. >>> >>> Nope. The checksums are part of the blockpointer, and the root >>> blockpointer is in the uberblock, which itself resides in the label. By >>> overwriting the label you've essentially erased all hope of practically >>> finding the root of the filesystem tree - not even checksumming all >>> possible block combinations (of which there are quite a few) will help >>> you here, because you have no checksums to compare them against. >>> >>> I'd love to be wrong, and I might be (I don't have as intimate a >>> knowledge of ZFS' on-disk structure as I'd like), but from where I'm >>> standing, your raidz vdev is essentially lost. >> >> The labels are not identical, because each contains the guid for the device. >> It is possible, though nontrivial, to recreate. >> >> That said, I've never seen a failure that just takes out only the ZFS labels. > > You'd have to go out of your way to take out the labels. Which is just what > I did (imagine: moving drives over to USB external enclosures, then putting > them onto a HP Raid controller (which overwrites the end of the disk) - which > also assumed that two disks should be automatically mirrored (if you miss the > 5 second prompt where you can tell it not to). ouch. But that shouldn't be enough. > Then try and recover the labels without really knowing what you're doing (my > bad). d'oh! > Suffice to say I have no confidence in the labels of two drives. On OI I can > forcefully import the pool but with any file that lives on multiple disks (ie, > over a certain size), all I get is an I/O error. Some of datasets also fail > to mount. please tell me you imported readonly? -- richard > > Thanks everyone for your input. > >> -- richard >> >> -- >> ZFS Performance and Training >> richard.ell...@richardelling.com >> +1-760-896-4422 >> >> >> >> >> >> >> -- ZFS Performance and Training richard.ell...@richardelling.com +1-760-896-4422 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Recovering lost labels on raidz member
On Mon, Aug 13, 2012 at 10:40:45AM -0700, Richard Elling wrote: > > On Aug 13, 2012, at 2:24 AM, Sa?o Kiselkov wrote: > > > On 08/13/2012 10:45 AM, Scott wrote: > >> Hi Saso, > >> > >> thanks for your reply. > >> > >> If all disks are the same, is the root pointer the same? > > > > No. > > > >> Also, is there a "signature" or something unique to the root block that I > >> can > >> search for on the disk? I'm going through the On-disk specification at the > >> moment. > > > > Nope. The checksums are part of the blockpointer, and the root > > blockpointer is in the uberblock, which itself resides in the label. By > > overwriting the label you've essentially erased all hope of practically > > finding the root of the filesystem tree - not even checksumming all > > possible block combinations (of which there are quite a few) will help > > you here, because you have no checksums to compare them against. > > > > I'd love to be wrong, and I might be (I don't have as intimate a > > knowledge of ZFS' on-disk structure as I'd like), but from where I'm > > standing, your raidz vdev is essentially lost. > > The labels are not identical, because each contains the guid for the device. > It is possible, though nontrivial, to recreate. > > That said, I've never seen a failure that just takes out only the ZFS labels. You'd have to go out of your way to take out the labels. Which is just what I did (imagine: moving drives over to USB external enclosures, then putting them onto a HP Raid controller (which overwrites the end of the disk) - which also assumed that two disks should be automatically mirrored (if you miss the 5 second prompt where you can tell it not to). Then try and recover the labels without really knowing what you're doing (my bad). Suffice to say I have no confidence in the labels of two drives. On OI I can forcefully import the pool but with any file that lives on multiple disks (ie, over a certain size), all I get is an I/O error. Some of datasets also fail to mount. Thanks everyone for your input. > -- richard > > -- > ZFS Performance and Training > richard.ell...@richardelling.com > +1-760-896-4422 > > > > > > > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Recovering lost labels on raidz member
On Aug 13, 2012, at 2:24 AM, Sašo Kiselkov wrote: > On 08/13/2012 10:45 AM, Scott wrote: >> Hi Saso, >> >> thanks for your reply. >> >> If all disks are the same, is the root pointer the same? > > No. > >> Also, is there a "signature" or something unique to the root block that I can >> search for on the disk? I'm going through the On-disk specification at the >> moment. > > Nope. The checksums are part of the blockpointer, and the root > blockpointer is in the uberblock, which itself resides in the label. By > overwriting the label you've essentially erased all hope of practically > finding the root of the filesystem tree - not even checksumming all > possible block combinations (of which there are quite a few) will help > you here, because you have no checksums to compare them against. > > I'd love to be wrong, and I might be (I don't have as intimate a > knowledge of ZFS' on-disk structure as I'd like), but from where I'm > standing, your raidz vdev is essentially lost. The labels are not identical, because each contains the guid for the device. It is possible, though nontrivial, to recreate. That said, I've never seen a failure that just takes out only the ZFS labels. -- richard -- ZFS Performance and Training richard.ell...@richardelling.com +1-760-896-4422 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] How do I import a zpool with a file as a member device?
On 08/13/2012 02:01 PM, Ray Arachelian wrote:
> On 08/13/2012 06:50 AM, Sašo Kiselkov wrote:
>> See the -d option to zpool import. -- Saso
>
> Many thanks for this, it worked very nicely, though the first time
> I ran it, it failed. So what -d does is to substitute /dev. In
> order for it to work, you also have to make links to the drive
> devices in the /dev tree.
>
> I did this in the directory where the file-members live (not
> knowing exactly off the top of my head which types of
> devices/slices zfs wanted) and then I was able to mount the pool,
> since the physical devices were c22t0d4-d7:
>
> for i in d4 d5 d6 d7; do for dev in /dev/dsk/c22t0${i}*; do echo
> $dev; ln -s $dev; done; done
>
> After this, using the -d option worked beautifully.
>
> Thanks for helping me recover my data. :)
>
No problem. Btw, the substitution for /dev/dsk is documented in the
manpage. But glad it worked. Enjoy ;-)
Cheers,
--
Saso
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] How do I import a zpool with a file as a member device?
On 08/13/2012 06:50 AM, Sašo Kiselkov wrote:
> See the -d option to zpool import. -- Saso
Many thanks for this, it worked very nicely, though the first time I ran
it, it failed. So what -d does is to substitute /dev. In order for it
to work, you also have to make links to the drive devices in the /dev tree.
I did this in the directory where the file-members live (not knowing
exactly off the top of my head which types of devices/slices zfs wanted)
and then I was able to mount the pool, since the physical devices were
c22t0d4-d7:
for i in d4 d5 d6 d7; do for dev in /dev/dsk/c22t0${i}*; do echo $dev;
ln -s $dev; done; done
After this, using the -d option worked beautifully.
Thanks for helping me recover my data. :)
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] How do I import a zpool with a file as a member device?
On 08/13/2012 12:48 PM, Ray Arachelian wrote: > While attempting to fix the last of my damaged zpools, there's one that > consists of 4 drives + one 60G file. The file happened by accident - I > attempted to add a partition off an SSD drive but missed the cache > keyword. Of course, once this is done, there's no way to remove the new > unwanted member. I fixed this by creating a file with mkfile, and > adding the file as a mirror of the SSD partition, then removed the SSD > partition from the zpool and kept the file around. > > Over time something went wrong with this zpool and it hung the OS from > booting, so I removed it from the machine, but held on to it so I could > attempt to recover. > > I'm trying to recover it now, but when I try to import it without cache > and read only, it tells me a member is missing (the file of course). > But, how do I pass the file name as a member of the zpool to the import > command? > > Since it's only 60g, I suppose I could dd the file over to an external > USB drive. Would that work? Do I have to write it to slice 0, or a > whole volume (slice 2)? I'm guessing this might be the path of least > resistance... > > Or is there some hidden option to zpool import where you can pass it > file/device names that I'm no aware of? (Attempting to use zpool online > poolname /path/to/file doesn't work since the zpool isn't imported yet. > Catch 22 there.) :) > > Otherwise, I do have a backup of the /etc/zfs/zpool.cache that contains > references to this zpool, but it seems that this is corrupted and > prevents all zpool operations. I've tried to replace the zpool members, > but it won't let me. I've even used a hex editor to replace the device > names in the zpool.cache file with the new device names, but it's still > locked. So I can't use the zpool cache file as a way to point to the > file member. Is there some way to clear the "corrupted" flag in the > zpool.cache from allowing zpool commands to work? > > I'm on oi151a5, but this zpool was created around oi134/135 or so. See the -d option to zpool import. -- Saso ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] How do I import a zpool with a file as a member device?
While attempting to fix the last of my damaged zpools, there's one that consists of 4 drives + one 60G file. The file happened by accident - I attempted to add a partition off an SSD drive but missed the cache keyword. Of course, once this is done, there's no way to remove the new unwanted member. I fixed this by creating a file with mkfile, and adding the file as a mirror of the SSD partition, then removed the SSD partition from the zpool and kept the file around. Over time something went wrong with this zpool and it hung the OS from booting, so I removed it from the machine, but held on to it so I could attempt to recover. I'm trying to recover it now, but when I try to import it without cache and read only, it tells me a member is missing (the file of course). But, how do I pass the file name as a member of the zpool to the import command? Since it's only 60g, I suppose I could dd the file over to an external USB drive. Would that work? Do I have to write it to slice 0, or a whole volume (slice 2)? I'm guessing this might be the path of least resistance... Or is there some hidden option to zpool import where you can pass it file/device names that I'm no aware of? (Attempting to use zpool online poolname /path/to/file doesn't work since the zpool isn't imported yet. Catch 22 there.) :) Otherwise, I do have a backup of the /etc/zfs/zpool.cache that contains references to this zpool, but it seems that this is corrupted and prevents all zpool operations. I've tried to replace the zpool members, but it won't let me. I've even used a hex editor to replace the device names in the zpool.cache file with the new device names, but it's still locked. So I can't use the zpool cache file as a way to point to the file member. Is there some way to clear the "corrupted" flag in the zpool.cache from allowing zpool commands to work? I'm on oi151a5, but this zpool was created around oi134/135 or so. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Recovering lost labels on raidz member
Thanks again Saso, at least I have closure :) Scott On Mon, Aug 13, 2012 at 11:24:55AM +0200, Sa?o Kiselkov wrote: > On 08/13/2012 10:45 AM, Scott wrote: > > Hi Saso, > > > > thanks for your reply. > > > > If all disks are the same, is the root pointer the same? > > No. > > > Also, is there a "signature" or something unique to the root block that I > > can > > search for on the disk? I'm going through the On-disk specification at the > > moment. > > Nope. The checksums are part of the blockpointer, and the root > blockpointer is in the uberblock, which itself resides in the label. By > overwriting the label you've essentially erased all hope of practically > finding the root of the filesystem tree - not even checksumming all > possible block combinations (of which there are quite a few) will help > you here, because you have no checksums to compare them against. > > I'd love to be wrong, and I might be (I don't have as intimate a > knowledge of ZFS' on-disk structure as I'd like), but from where I'm > standing, your raidz vdev is essentially lost. > > -- > Saso > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Recovering lost labels on raidz member
On 08/13/2012 10:45 AM, Scott wrote: > Hi Saso, > > thanks for your reply. > > If all disks are the same, is the root pointer the same? No. > Also, is there a "signature" or something unique to the root block that I can > search for on the disk? I'm going through the On-disk specification at the > moment. Nope. The checksums are part of the blockpointer, and the root blockpointer is in the uberblock, which itself resides in the label. By overwriting the label you've essentially erased all hope of practically finding the root of the filesystem tree - not even checksumming all possible block combinations (of which there are quite a few) will help you here, because you have no checksums to compare them against. I'd love to be wrong, and I might be (I don't have as intimate a knowledge of ZFS' on-disk structure as I'd like), but from where I'm standing, your raidz vdev is essentially lost. -- Saso ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Recovering lost labels on raidz member
Hi Saso, thanks for your reply. If all disks are the same, is the root pointer the same? Also, is there a "signature" or something unique to the root block that I can search for on the disk? I'm going through the On-disk specification at the moment. Scott On Mon, Aug 13, 2012 at 10:02:58AM +0200, Sa?o Kiselkov wrote: > On 08/13/2012 10:00 AM, Sa?o Kiselkov wrote: > > On 08/13/2012 03:02 AM, Scott wrote: > >> Hi all, > >> > >> I have a 5 disk raidz array in a state of disrepair. Suffice to say three > >> disks are ok, while two are missing all their labels. (Both ends of the > >> disks were overwritten). The data is still intact. > > > > There are 4 labels on a zfs-labeled disk, two at the start and two at > > the end. Have all been overwritten? > > Just re-read your post again, and I realized my question here is > redundant. Without the labels your data is toast. > > -- > Saso > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Recovering lost labels on raidz member
On 08/13/2012 10:00 AM, Sašo Kiselkov wrote: > On 08/13/2012 03:02 AM, Scott wrote: >> Hi all, >> >> I have a 5 disk raidz array in a state of disrepair. Suffice to say three >> disks are ok, while two are missing all their labels. (Both ends of the >> disks were overwritten). The data is still intact. > > There are 4 labels on a zfs-labeled disk, two at the start and two at > the end. Have all been overwritten? Just re-read your post again, and I realized my question here is redundant. Without the labels your data is toast. -- Saso ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Recovering lost labels on raidz member
On 08/13/2012 03:02 AM, Scott wrote: > Hi all, > > I have a 5 disk raidz array in a state of disrepair. Suffice to say three > disks are ok, while two are missing all their labels. (Both ends of the > disks were overwritten). The data is still intact. There are 4 labels on a zfs-labeled disk, two at the start and two at the end. Have all been overwritten? > Unfortunately I don't have a zpool.cache either. > > Is there a way to reconstruct the labels using the infomration from the 3 > valid disks? The simple answer is: no, without labels those disks are toast. The more complicated answer: the labels contain the root block pointer to the root of the filesystem tree on the given device. Without it you don't know where to start looking for many of the top-level structures needed to begin making sense of the data on the disk. -- Saso ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
