[zfs-discuss] zpool recovery import from dd images
I am in a rather unique situation. I've inherited a zpool composed of two vdevs. One vdev was roughly 9TB on one RAID 5 array, and the other vdev is roughly 2TB on a different RAID 5 array.The 9TB array crashed and was sent to a data recovery firm, and they've given me a dd image. I've also taken a dd image of the other side. -rw-r--r-- 1 root wheel 9.2T Aug 18 07:53 deep_Lun0.dd -rw-r--r-- 1 root wheel 2.1T Aug 23 15:55 deep_san01_lun.dd zpool import identifies that both files are members of the pool, but there are 'insufficient replicas' and it can not import. zpool import -d . -F pool: deep id: 8026270260630449340 state: UNAVAIL status: The pool was last accessed by another system. action: The pool cannot be imported due to damaged devices or data. see: http://www.sun.com/msg/ZFS-8000-EY config: deep UNAVAIL insufficient replicas /jbod1-diskbackup/restore/deep_san01_lun.dd UNAVAIL cannot open /jbod1-diskbackup/restore/deep_Lun0.dd UNAVAIL cannot open Using zdb, I can see both images have 4 labels. What are my options in terms of attempting a recovery? I had hoped that ZFS could import the pool even if the vdevs are now files instead of devices, but unfortunately it was not so easy. zdb -l deep_Lun0.dd LABEL 0 version: 15 name: 'deep' state: 0 txg: 322458504 pool_guid: 8026270260630449340 hostid: 713828554 hostname: 'vali.REDACTED top_guid: 7224951874042153155 guid: 7224951874042153155 vdev_tree: type: 'disk' id: 1 guid: 7224951874042153155 path: '/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0' devid: 'id1,sd@TRaidWeb.Com_003089320001061_/a' phys_path: '/scsi_vhci/disk@g526169645765622e436f6d202020202030303330383933323030303130363120:a' whole_disk: 0 metaslab_array: 148 metaslab_shift: 36 ashift: 9 asize: 10076008480768 is_log: 0 DTL: 213 LABEL 1 version: 15 name: 'deep' state: 0 txg: 322458504 pool_guid: 8026270260630449340 hostid: 713828554 hostname: 'vali.REDACTED top_guid: 7224951874042153155 guid: 7224951874042153155 vdev_tree: type: 'disk' id: 1 guid: 7224951874042153155 path: '/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0' devid: 'id1,sd@TRaidWeb.Com_003089320001061_/a' phys_path: '/scsi_vhci/disk@g526169645765622e436f6d202020202030303330383933323030303130363120:a' whole_disk: 0 metaslab_array: 148 metaslab_shift: 36 ashift: 9 asize: 10076008480768 is_log: 0 DTL: 213 LABEL 2 version: 15 name: 'deep' state: 0 txg: 322458504 pool_guid: 8026270260630449340 hostid: 713828554 hostname: 'vali.REDACTED top_guid: 7224951874042153155 guid: 7224951874042153155 vdev_tree: type: 'disk' id: 1 guid: 7224951874042153155 path: '/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0' devid: 'id1,sd@TRaidWeb.Com_003089320001061_/a' phys_path: '/scsi_vhci/disk@g526169645765622e436f6d202020202030303330383933323030303130363120:a' whole_disk: 0 metaslab_array: 148 metaslab_shift: 36 ashift: 9 asize: 10076008480768 is_log: 0 DTL: 213 LABEL 3 version: 15 name: 'deep' state: 0 txg: 322458504 pool_guid: 8026270260630449340 hostid: 713828554 hostname: 'vali.REDACTED top_guid: 7224951874042153155 guid: 7224951874042153155 vdev_tree: type: 'disk' id: 1 guid: 7224951874042153155 path: '/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0' devid: 'id1,sd@TRaidWeb.Com_003089320001061_/a' phys_path: '/scsi_vhci/disk@g526169645765622e436f6d202020202030303330383933323030303130363120:a' whole_disk: 0 metaslab_array: 148 metaslab_shift: 36 ashift: 9 asize: 10076008480768 is_log: 0 DTL: 213 zdb -l deep_san01_lun.dd LABEL 0 version: 15 name: 'deep' state: 0 txg: 322458504 pool_guid: 8026270260630449340 hostid: 713828554 hostname: 'vali.REDACTED top_guid: 6101030958963841536 guid: 6101030958963841536 vdev_tree:
Re: [zfs-discuss] zpool import starves machine of memory
UPDATE (for those following along at home)...
After patching to latest and greatest Solaris 10 kernel update
and getting firmware on both OS drives (72 GB SAS) and server updated
to latest and greatest, Oracle has now officially declared it a bug
(CR#7082249). No word on when I'll hear back on status of this new bug
(which looks like an old bug, but the old bug has been fixed in the
patches I'm now running).
On Wed, Aug 3, 2011 at 9:19 AM, Paul Kraus p...@kraus-haus.org wrote:
I am having a very odd problem, and so far the folks at Oracle
Support have not provided a working solution, so I am asking the crowd
here while still pursuing it via Oracle Support.
The system is a T2000 running 10U9 with CPU-2010-01and two J4400
loaded with 1 TB SATA drives. There is one zpool on the J4400 (3 x 15
disk vdev + 3 hot spare). This system is the target for zfs send /
recv replication from our production server.The OS is UFS on local
disk.
snip
--
{1-2-3-4-5-6-7-}
Paul Kraus
- Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
- Sound Designer: Frankenstein, A New Musical
(http://www.facebook.com/event.php?eid=123170297765140)
- Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
- Technical Advisor, RPI Players
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool recovery import from dd images
Hi Kelsey, I haven't had to do this myself so someone who has done this before might have a better suggestion. I wonder if you need to make links from the original device name to the new device names. You can see from the zdb -l output below that the device path is pointing to the original device names (really long device names). Thanks, Cindy On 08/24/11 12:11, Kelsey Damas wrote: I am in a rather unique situation. I've inherited a zpool composed of two vdevs. One vdev was roughly 9TB on one RAID 5 array, and the other vdev is roughly 2TB on a different RAID 5 array.The 9TB array crashed and was sent to a data recovery firm, and they've given me a dd image. I've also taken a dd image of the other side. -rw-r--r-- 1 root wheel 9.2T Aug 18 07:53 deep_Lun0.dd -rw-r--r-- 1 root wheel 2.1T Aug 23 15:55 deep_san01_lun.dd zpool import identifies that both files are members of the pool, but there are 'insufficient replicas' and it can not import. zpool import -d . -F pool: deep id: 8026270260630449340 state: UNAVAIL status: The pool was last accessed by another system. action: The pool cannot be imported due to damaged devices or data. see: http://www.sun.com/msg/ZFS-8000-EY config: deep UNAVAIL insufficient replicas /jbod1-diskbackup/restore/deep_san01_lun.dd UNAVAIL cannot open /jbod1-diskbackup/restore/deep_Lun0.dd UNAVAIL cannot open Using zdb, I can see both images have 4 labels. What are my options in terms of attempting a recovery? I had hoped that ZFS could import the pool even if the vdevs are now files instead of devices, but unfortunately it was not so easy. zdb -l deep_Lun0.dd LABEL 0 version: 15 name: 'deep' state: 0 txg: 322458504 pool_guid: 8026270260630449340 hostid: 713828554 hostname: 'vali.REDACTED top_guid: 7224951874042153155 guid: 7224951874042153155 vdev_tree: type: 'disk' id: 1 guid: 7224951874042153155 path: '/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0' devid: 'id1,sd@TRaidWeb.Com_003089320001061_/a' phys_path: '/scsi_vhci/disk@g526169645765622e436f6d202020202030303330383933323030303130363120:a' whole_disk: 0 metaslab_array: 148 metaslab_shift: 36 ashift: 9 asize: 10076008480768 is_log: 0 DTL: 213 LABEL 1 version: 15 name: 'deep' state: 0 txg: 322458504 pool_guid: 8026270260630449340 hostid: 713828554 hostname: 'vali.REDACTED top_guid: 7224951874042153155 guid: 7224951874042153155 vdev_tree: type: 'disk' id: 1 guid: 7224951874042153155 path: '/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0' devid: 'id1,sd@TRaidWeb.Com_003089320001061_/a' phys_path: '/scsi_vhci/disk@g526169645765622e436f6d202020202030303330383933323030303130363120:a' whole_disk: 0 metaslab_array: 148 metaslab_shift: 36 ashift: 9 asize: 10076008480768 is_log: 0 DTL: 213 LABEL 2 version: 15 name: 'deep' state: 0 txg: 322458504 pool_guid: 8026270260630449340 hostid: 713828554 hostname: 'vali.REDACTED top_guid: 7224951874042153155 guid: 7224951874042153155 vdev_tree: type: 'disk' id: 1 guid: 7224951874042153155 path: '/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0' devid: 'id1,sd@TRaidWeb.Com_003089320001061_/a' phys_path: '/scsi_vhci/disk@g526169645765622e436f6d202020202030303330383933323030303130363120:a' whole_disk: 0 metaslab_array: 148 metaslab_shift: 36 ashift: 9 asize: 10076008480768 is_log: 0 DTL: 213 LABEL 3 version: 15 name: 'deep' state: 0 txg: 322458504 pool_guid: 8026270260630449340 hostid: 713828554 hostname: 'vali.REDACTED top_guid: 7224951874042153155 guid: 7224951874042153155 vdev_tree: type: 'disk' id: 1 guid: 7224951874042153155 path: '/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0' devid: 'id1,sd@TRaidWeb.Com_003089320001061_/a' phys_path: '/scsi_vhci/disk@g526169645765622e436f6d202020202030303330383933323030303130363120:a' whole_disk: 0 metaslab_array: 148 metaslab_shift: 36 ashift: 9 asize: 10076008480768
Re: [zfs-discuss] zpool recovery import from dd images
On Wed, Aug 24, 2011 at 1:23 PM, Cindy Swearingen cindy.swearin...@oracle.com wrote: I wonder if you need to make links from the original device name to the new device names. You can see from the zdb -l output below that the device path is pointing to the original device names (really long device names). Thank you for this suggestion. I've created symlinks from the /dev/dsk directory pointing to the dd image, but zpool import says the same thing. However, look at the error message below. zpool can see the hostname of the last system to use the pool, but the date looks like the epoch (Dec 31 1969). Is this meaningful? # ln -s /jbod1-diskbackup/restore/deep_Lun0.dd /dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0 # ln -s /jbod1-diskbackup/restore/deep_san01_lun.dd /dev/dsk/c4t526169645765622E436F6D202020202030303131383933323030303130354220d0s0 # zpool import -d . pool: deep id: 8026270260630449340 state: UNAVAIL status: The pool was last accessed by another system. action: The pool cannot be imported due to damaged devices or data. see: http://www.sun.com/msg/ZFS-8000-EY config: deep UNAVAIL insufficient replicas /jbod1-diskbackup/restore/deep_san01_lun.dd UNAVAIL cannot open /jbod1-diskbackup/restore/deep_Lun0.dd UNAVAIL cannot open # zpool import -d . -F deep cannot import 'deep': pool may be in use from other system, it was last accessed by vali.REDACTED (hostid: 0x2a8c28ca) on Wed Dec 31 16:00:00 1969 use '-f' to import anyway # zpool import -d . -F -f deep cannot import 'deep': one or more devices is currently unavailable ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool recovery import from dd images
On 24 August, 2011 - Kelsey Damas sent me these 1,8K bytes:
On Wed, Aug 24, 2011 at 1:23 PM, Cindy Swearingen
cindy.swearin...@oracle.com wrote:
I wonder if you need to make links from the original device
name to the new device names.
You can see from the zdb -l output below that the device path
is pointing to the original device names (really long device
names).
Thank you for this suggestion. I've created symlinks from the
/dev/dsk directory pointing to the dd image, but zpool import says the
same thing. However, look at the error message below. zpool can
see the hostname of the last system to use the pool, but the date
looks like the epoch (Dec 31 1969). Is this meaningful?
# ln -s /jbod1-diskbackup/restore/deep_Lun0.dd
/dev/dsk/c4t526169645765622E436F6D202020202030303330383933323030303130363120d0s0
# ln -s /jbod1-diskbackup/restore/deep_san01_lun.dd
/dev/dsk/c4t526169645765622E436F6D202020202030303131383933323030303130354220d0s0
# zpool import -d .
Just for fun, try an absolute path.
/Tomas
--
Tomas Forsman, st...@acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Umeå
`- Sysadmin at {cs,acc}.umu.se
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool recovery import from dd images
Just for fun, try an absolute path. Thank you again for the suggestions. I was able to make this work with lofiadm to mount the images. Then, be sure to give zpool the -d flag to scan /dev/lofi # lofiadm -a /jbod1-diskbackup/restore/deep_Lun0.dd /dev/lofi/1 # lofiadm -a /jbod1-diskbackup/restore/deep_san01_lun.dd /dev/lofi/2 # zpool import -d /dev/lofi pool: deep id: 8026270260630449340 state: ONLINE status: The pool was last accessed by another system. action: The pool can be imported using its name or numeric identifier and the '-f' flag. see: http://www.sun.com/msg/ZFS-8000-EY config: deep ONLINE /dev/lofi/2 ONLINE /dev/lofi/1 ONLINE ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpool recovery import from dd images
markm wrote: Because the vdev tree is calling them 'disk', zfs is attempting to open them using disk i/o instead of file i/o. This was correct, thank you. lofiadm was useful to loopback mount the image files to provide disk i/o. ZFS has much more opportunity to recover from device failure when it has a redundant config. Splitting the 9T 2T LUNs into a few separate LUNs and then using raidz would be highly desirable. Yes, I completely agree. This configuration is something I inherited from a previous colleague, and did not make good use of ZFS.The underlying RAID system suffered a catastrophic failure, but even prior to that, we would consistently have the pools become DEGRADED when ZFS would report corruption that the underlying RAID system could not detect and repair. Thank you to all responders, and also thank you to Drive Savers who recovered the image for us. If you find yourself in the regrettable situation of having a dozen terabytes go missing one day without proper backups, I highly recommend their services. -- Kelsey ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
