Re: [zfs-discuss] problem with zpool import - zil and cache drive are not displayed?
On Aug 4, 2010, at 7:15 AM, Dmitry Sorokin wrote: I'm in the same situation as Darren - my log SSD device died completely. Victor, could you please explain how did you mocked up log device in a file so zpool status started to show the device with UNAVAIL status? I lost the latest zpool.cache file, but I was able to recover GUID of the log device from the backup copy of zpool.cache. Well, that's not very difficult. You need to write proper VDEV configuration with good checksum into at least one ZFS label of some kind of new device - either disk of file. IF you have backup zpool.cache with necessary details then is it not that difficult. Btw, in Darren's case we almost succeeded - it was possible to import pool with mocked up log device, but due to corruption in metaslabs it panicked almost immediately. For some reason setting aok/zfs_recover did not help too. Last option was to try readonly import but I was not able to prepare necessary bits quickly enough and Darren decided to stop pursuing recovery and revert to partial backups he had. I'm almost sure that readonly import would let him get everything back. In future it should be easier as ZFS readonly import support is now integrated into source code thanks to George Wilson's efforts. regards victor Thanks, Dmitry -Original Message- From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-boun...@opensolaris.org] On Behalf Of Victor Latushkin Sent: Tuesday, August 03, 2010 7:09 PM To: Darren Taylor Cc: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] problem with zpool import - zil and cache drive are not displayed? On Aug 4, 2010, at 12:23 AM, Darren Taylor wrote: Hi George, I think you are right. The log device looks to have suffered a complete loss, there is no data on the disk at all. The log device was a acard ram drive (with battery backup), but somehow it has faulted clearing all data. --victor gave me this advice, and queried about the zpool.cache-- Looks like there's a hardware problem with c7d0 as it appears to contain garbage. Do you have zpool.cache with this pool configuration available? Besides containing garbage former log device now appears to have different geometry and is not able to read in the higher LBA ranges. So i'd say it is broken. c7d0 was the log device. I'm unsure what the next step is, but i'm assuming there is a way to grab the drives original config from the zpool.cache file and apply back to the drive? I mocked up log device in a file, and that made zpool import more happy: bash-4.0# zpool import pool: tank id: 15136317365944618902 state: DEGRADED status: The pool was last accessed by another system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: http://www.sun.com/msg/ZFS-8000-EY config: tankDEGRADED raidz1-0 ONLINE c6t4d0 ONLINE c6t5d0 ONLINE c6t6d0 ONLINE c6t7d0 ONLINE raidz1-1 ONLINE c6t0d0 ONLINE c6t1d0 ONLINE c6t2d0 ONLINE c6t3d0 ONLINE cache c8d1 logs c13d1s0 UNAVAIL cannot open bash-4.0# zpool import -fR / tank cannot import 'tank': one or more devices is currently unavailable Recovery is possible, but will result in some data loss. Returning the pool to its state as of July 21, 2010 03:49:50 AM NZST should correct the problem. Approximately 91 seconds of data must be discarded, irreversibly. After rewind, several persistent user-data errors will remain. Recovery can be attempted by executing 'zpool import -F tank'. A scrub of the pool is strongly recommended after recovery. bash-4.0# So if you are happy with the results, you can perform actual import with zpool import -fF -R / tank You should then be able to remove log device completely. regards victor ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] problem with zpool import - zil and cache drive are not displayed?
zpool import -fF -R / tank As above victor managed to mock up a log device and get the pool to a state where it could be imported again. Awesome stuff. I'm also interested in how if its not too complex. :) Unfortunately though it looks like something is still not quite right. Even though running zpool import -fR / tank seems to suggest it is going to recovery and loose 91 seconds when i actually run the command zpool import -fF -R / tank the machine locks up and reboots. I've tried this multiple times with the same results. h. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] problem with zpool import - zil and cache drive are not displayed?
I'm at a loss, I've managed to get myself into a fix. I'm not sure where the problem is, but essentially i have a zpool i cannot import. This particular pool used to have a two drives (not shown below), one for cache and another for log. I'm unsure why they are no longer detected on zpool import... the disks are still connected to the system and show up when running format for a list. dar...@lexx:~# zpool import pool: tank id: 15136317365944618902 state: UNAVAIL status: The pool was last accessed by another system. action: The pool cannot be imported due to damaged devices or data. see: http://www.sun.com/msg/ZFS-8000-EY config: tank UNAVAIL missing device raidz1-0 ONLINE c6t4d0 ONLINE c6t5d0 ONLINE c6t6d0 ONLINE c6t7d0 ONLINE raidz1-1 ONLINE c6t0d0 ONLINE c6t1d0 ONLINE c6t2d0 ONLINE c6t3d0 ONLINE dar...@lexx:~# The above disks are the data disks which appear to be online without issue. i was running version 22 on this pool. Any help appreciated -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] problem with zpool import - zil and cache drive are not displayed?
Darren, It looks like you've lost your log device. The newly integrated missing log support will help once it's available. In the meantime, you should run 'zdb -l' on your log device to make sure the label is still intact. Thanks, George Darren Taylor wrote: I'm at a loss, I've managed to get myself into a fix. I'm not sure where the problem is, but essentially i have a zpool i cannot import. This particular pool used to have a two drives (not shown below), one for cache and another for log. I'm unsure why they are no longer detected on zpool import... the disks are still connected to the system and show up when running format for a list. dar...@lexx:~# zpool import pool: tank id: 15136317365944618902 state: UNAVAIL status: The pool was last accessed by another system. action: The pool cannot be imported due to damaged devices or data. see: http://www.sun.com/msg/ZFS-8000-EY config: tank UNAVAIL missing device raidz1-0 ONLINE c6t4d0 ONLINE c6t5d0 ONLINE c6t6d0 ONLINE c6t7d0 ONLINE raidz1-1 ONLINE c6t0d0 ONLINE c6t1d0 ONLINE c6t2d0 ONLINE c6t3d0 ONLINE dar...@lexx:~# The above disks are the data disks which appear to be online without issue. i was running version 22 on this pool. Any help appreciated ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] problem with zpool import - zil and cache drive are not displayed?
On Aug 4, 2010, at 12:23 AM, Darren Taylor wrote: Hi George, I think you are right. The log device looks to have suffered a complete loss, there is no data on the disk at all. The log device was a acard ram drive (with battery backup), but somehow it has faulted clearing all data. --victor gave me this advice, and queried about the zpool.cache-- Looks like there's a hardware problem with c7d0 as it appears to contain garbage. Do you have zpool.cache with this pool configuration available? Besides containing garbage former log device now appears to have different geometry and is not able to read in the higher LBA ranges. So i'd say it is broken. c7d0 was the log device. I'm unsure what the next step is, but i'm assuming there is a way to grab the drives original config from the zpool.cache file and apply back to the drive? I mocked up log device in a file, and that made zpool import more happy: bash-4.0# zpool import pool: tank id: 15136317365944618902 state: DEGRADED status: The pool was last accessed by another system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: http://www.sun.com/msg/ZFS-8000-EY config: tankDEGRADED raidz1-0 ONLINE c6t4d0 ONLINE c6t5d0 ONLINE c6t6d0 ONLINE c6t7d0 ONLINE raidz1-1 ONLINE c6t0d0 ONLINE c6t1d0 ONLINE c6t2d0 ONLINE c6t3d0 ONLINE cache c8d1 logs c13d1s0 UNAVAIL cannot open bash-4.0# zpool import -fR / tank cannot import 'tank': one or more devices is currently unavailable Recovery is possible, but will result in some data loss. Returning the pool to its state as of July 21, 2010 03:49:50 AM NZST should correct the problem. Approximately 91 seconds of data must be discarded, irreversibly. After rewind, several persistent user-data errors will remain. Recovery can be attempted by executing 'zpool import -F tank'. A scrub of the pool is strongly recommended after recovery. bash-4.0# So if you are happy with the results, you can perform actual import with zpool import -fF -R / tank You should then be able to remove log device completely. regards victor ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] problem with zpool import - zil and cache drive are not displayed?
From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Darren Taylor I'm not sure where the problem is, but essentially i have a zpool i cannot import. This particular pool used to have a two drives (not shown below), one for cache and another for log. I'm unsure why they are no longer detected on zpool import... the disks are still connected to the system and show up when running format for a list. Perhaps the log cache were not using entire devices, but rather, just slices? I could be wrong, but I don't think zpool import will scan slices by default. If slices exist on the cache log devices, I might suggest using the -d option of zpool import. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] problem with zpool import - zil and cache drive are not displayed?
On Aug 3, 2010, at 8:39 PM, Edward Ned Harvey wrote: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Darren Taylor I'm not sure where the problem is, but essentially i have a zpool i cannot import. This particular pool used to have a two drives (not shown below), one for cache and another for log. I'm unsure why they are no longer detected on zpool import... the disks are still connected to the system and show up when running format for a list. Perhaps the log cache were not using entire devices, but rather, just slices? I could be wrong, but I don't think zpool import will scan slices by default. Entire devices do not exist, only slices. If slices exist on the cache log devices, I might suggest using the -d option of zpool import. The -d option allows searching in another directory. -- richard -- Richard Elling rich...@nexenta.com +1-760-896-4422 Enterprise class storage for everyone www.nexenta.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
