Re: [zfs-discuss] Directory is not accessible
On 2012-11-26 15:15, The OP wrote: How can one remove a directory containing corrupt files or a corrupt file itself? For me rm just gives input/output error. I believe you can get rid of the corrupt files by overwriting them. In my case of corrupted files, I dd'ed the corrupt blocks from a backup source into the right spot of the file. Overall this released the corrupt blocks from the pool and allowed them to get freed (or perhaps leaked in case of that bug I've stepped onto). Trying to free the block can get your pool into trouble or panics, depending on the nature of the corruption, though (in my case, DDT was trying to release a block that was not entered into the DDT). If this happens, your next best bet would be to trace where the error happens, invent a patch (such as letting it possibly leak away) and compile your own kernel to clean up the pool. Of course, it is also possible that the block would go away (if it is not referenced also by snapshots/clones/dedup), and such drastic measures won't be needed. HTH, //Jim Klimov ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Directory is not accessible
unlink(1M)? cheers, --justin From: Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) To: Sami Tuominen ; " zfs-discuss@opensolaris.org" Sent: Monday, 26 November 2012, 14:57 Subject: Re: [zfs-discuss] Directory is not accessible > From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Sami Tuominen > > How can one remove a directory containing corrupt files or a corrupt file > itself? For me rm just gives input/output error. I was hoping to see somebody come up with an answer for this ... I would expect rm to work... Maybe you have to rm the parent of the thing you're trying to rm? But I kinda doubt it. Maybe you need to verify you're rm'ing the right thing? I believe, if you scrub the pool, it should tell you the name of the corrupt things. Or maybe you're not experiencing a simple cksum mismatch, maybe you're experiencing a legitimate IO error. The "rm" solution could only possibly work to clear up a cksum mismatch. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Directory is not accessible
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Sami Tuominen > > How can one remove a directory containing corrupt files or a corrupt file > itself? For me rm just gives input/output error. I was hoping to see somebody come up with an answer for this ... I would expect rm to work... Maybe you have to rm the parent of the thing you're trying to rm? But I kinda doubt it. Maybe you need to verify you're rm'ing the right thing? I believe, if you scrub the pool, it should tell you the name of the corrupt things. Or maybe you're not experiencing a simple cksum mismatch, maybe you're experiencing a legitimate IO error. The "rm" solution could only possibly work to clear up a cksum mismatch. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Directory is not accessible
How can one remove a directory containing corrupt files or a corrupt file itself? For me rm just gives input/output error. Sami "Edward Ned Harvey (opensolarisisdeadlongliveopensolaris) " wrote: > From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Sami Tuominen > > Unfortunately there aren't any snapshots. > The version of zpool is 15. Is it safe to upgrade that? > Is zpool clear -F supported or of any use here? The only thing that will be of use to restore your data will be a backup. To forget about the lost data and make the error message go away, simply rm the bad directory (and/or its parent). You're probably wondering, you have redundancy and no faulted devices, so how could this happen? There are a few possible explanations, but they're all going to have one thing in common: At some point, something got corrupted before it was written corrupted and the redundant copy also written corrupted. It might be you had a CPU error, or some parity error in non-ECC ram, or a bus glitch or bad firmware in the HBA, for example. The fact remains, something was written corrupted, and the redundant copy was also written corrupted. All you can do is restore from a snapshot, restore from a backup, or accept it for what it is and make the error go away. Sorry to hear it... ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Directory is not accessible
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Sami Tuominen > > Unfortunately there aren't any snapshots. > The version of zpool is 15. Is it safe to upgrade that? > Is zpool clear -F supported or of any use here? The only thing that will be of use to restore your data will be a backup. To forget about the lost data and make the error message go away, simply rm the bad directory (and/or its parent). You're probably wondering, you have redundancy and no faulted devices, so how could this happen? There are a few possible explanations, but they're all going to have one thing in common: At some point, something got corrupted before it was written corrupted and the redundant copy also written corrupted. It might be you had a CPU error, or some parity error in non-ECC ram, or a bus glitch or bad firmware in the HBA, for example. The fact remains, something was written corrupted, and the redundant copy was also written corrupted. All you can do is restore from a snapshot, restore from a backup, or accept it for what it is and make the error go away. Sorry to hear it... ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Directory is not accessible
>> >> NAMESTATE READ WRITE CKSUM >> tankki ONLINE 0 0 3.62K >> raidz1-0 ONLINE 0 0 14.5K >> ada5p2 ONLINE 0 0 0 >> ada2p2 ONLINE 0 0 0 >> ada4p2 ONLINE 0 0 0 >> ada3p2 ONLINE 0 0 0 >> ada0p2 ONLINE 0 0 0 >> ada1p2 ONLINE 0 0 0 >> >> errors: Permanent errors have been detected in the following files: >> >> tankki/media:<0x0> > > It's as it says it is: the error is "permanent" in that ZFS has done what it > could to recover the data > from parity information and ditto blocks. Sometimes the error is only in the > current version of a > file/directory, so you can recover the data from a snapshot. Unfortunately there aren't any snapshots. The version of zpool is 15. Is it safe to upgrade that? Is zpool clear -F supported or of any use here? Sami > Jan ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Directory is not accessible
On Sun, Oct 7, 2012 at 12:59 PM, Sami Tuominen wrote: > Hi > > I have raidz pool with one directory, which is not accessible. It only gives > "Input/output error" when trying to access it. Is there any way to fix that? [...] > nas4free:/tankki/media# zpool status -v > pool: tankki > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. >see: http://illumos.org/msg/ZFS-8000-8A > scan: scrub in progress since Sun Oct 7 21:18:19 2012 > 494G scanned out of 5.92T at 261M/s, 6h4m to go > 0 repaired, 8.15% done > config: > > NAMESTATE READ WRITE CKSUM > tankki ONLINE 0 0 3.62K > raidz1-0 ONLINE 0 0 14.5K > ada5p2 ONLINE 0 0 0 > ada2p2 ONLINE 0 0 0 > ada4p2 ONLINE 0 0 0 > ada3p2 ONLINE 0 0 0 > ada0p2 ONLINE 0 0 0 > ada1p2 ONLINE 0 0 0 > > errors: Permanent errors have been detected in the following files: > > tankki/media:<0x0> It's as it says it is: the error is "permanent" in that ZFS has done what it could to recover the data from parity information and ditto blocks. Sometimes the error is only in the current version of a file/directory, so you can recover the data from a snapshot. > nas4free:/tankki/media# cd Dokumentit > Dokumentit: Input/output error. > nas4free:/tankki/media# Do you have a snapshot that you can navigate to and determine if the directory appears intact? Jan ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
