Re: [zfs-discuss] Remove corrupt files from snapshot
On Tue, November 15, 2011 10:07, sbre...@hotmail.com wrote: > Would it make sense to do "zfs scrub" regularly and have a report sent, > i.e. once a day, so discrepancy would be noticed beforehand? Is there > anything readily available in the Freebsd ZFS package for this? If you're not scrubbing regularly, you're losing out on one of the key benefits of ZFS. In nearly all fileserver situations, a good amount of the content is essentially archival, infrequently accessed but important now and then. (In my case it's my collection of digital and digitized photos.) A weekly scrub combined with a decent backup plan will detect bit-rot before the backups with the correct data cycle into the trash (and, with redundant storage like mirroring or RAID, the scrub will probably be able to fix the error without resorting to restoring files from backup). -- David Dyer-Bennet, d...@dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Todd Urie > > Use zpool status -v to see if any errors come up. Then you can use zpool > scrub to remove at least some of them. I have had luck with this in the past. Disks are made of chemicals, which can degrade over time. If some part of a disk starts to deteriorate, but you never attempt to read it, then you'll never know it's going bad. You should have redundancy, and scrub on a regular basis, much more frequently than the occurrence of a disk going bad - maybe once a week or once a month. If you can afford to scrub daily, that's great. Depending on your system and your data, scrubs might take several hours, thus making it impractical to scrub daily. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
Use zpool status -v to see if any errors come up. Then you can use zpool scrub
to remove at least some of them. I have had luck with this in the past.
---Todd
On Nov 14, 2011, at 04:25 , wrote:
>
> Back to this topic, since I cannot touch snapshots I thought I could simply
> remove the corrupt files after the last snapshot, so the next incremental
> backup will notice the difference (i.e. no file) and overwrite the
> corrupt-and-removed files with valid ones. This was the plan.
>
> However, while checking for corrupt files, "find" stops at some directory
> with "fts_read: Not a directory":
>
> find . -exec md5 {} \; > /home/xxx/md5_out 2> /home/xxx/md5_err &
>
> tail /home/xxx/md5_err
> ...
> md5: ./.zfs/snapshot/20100323081201/Bazsi/Projects/Java Test
> Client/java_test_client/lib/xxx/weblogic.jar: Input/output error
> md5: ./.zfs/snapshot/20100323081201/@Cache (Bazsi)/BMWi
> SP/Publikationen/PDF-Brosch▒ren/Nexxt.pdf: Input/output error
> find: fts_read: Not a directory
>
> What does this error mean? I cannot even "scan" the ZFS file system anymore?
> Is there any "fsck" for ZFS?
>
>
> Cheers,
> B.
>
> ----
>> From: zfsdisc...@orgdotuk.org.uk
>> To: zfs-discuss@opensolaris.org
>> Date: Mon, 7 Nov 2011 21:49:56 +
>> Subject: Re: [zfs-discuss] Remove corrupt files from snapshot
>>
>>> -Original Message-
>>> From: Edward Ned Harvey
>>> Sent: 04/11/2011 21:23
>>>
>>> You need to destroy the snapshot completely - But if you want
>>> to selectively
>>> delete from a snapshot, I think you can clone it, then
>>> promote the clone,
>>> then destroy the snapshot, then rm something from the clone and then
>>> snapshot the clone back to the original name, and then
>>> destroy the clone.
>>>
>>> Right?
>>
>> Not so fast! :-)
>>
>> If you promote this new clone, the current state / branch of your filesystem
>> becomes a clone instead, dependent on the snapshot.
>> Then if you try to destroy the snapshot, you'll fail, because it has a
>> dependent clone (your current fs!!!). If you continue
>> without realising the implications, and so try the 'destroy' again with
>> '-R', there goes the neighbourhood!
>>
>> I did this once, and was only saved by the fact that my cwd was in my
>> current filesystem, so couldn't be unmounted, and therefore
>> couldn't be removed! Phew!! Nice to learn something and only get singed
>> eyebrows, instead of losing a leg!
>>
>> hth Andy
>>
>>
>> ___
>> zfs-discuss mailing list
>> zfs-discuss@opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>>
> ___
> zfs-discuss mailing list
> zfs-discuss@opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
On Tue, Nov 15, 2011 at 8:07 AM, wrote: > Thanks anyone for the help, finally I removed corrupt files from the > "current view" of the file system and left the snapshots as they were. This > way at least the incremental backup continues. (It is sad that snapshots > are so rigid that even corruption is permanent. What more interesting is > that, if snapshots are read only, how can they become corrupted?) > The snapshot is read-only, meaning users cannot modify the data in the snapshots. However, there's nothing to prevents random bit flips in the underlying storage. Maybe the physical harddrive has a bad block and gmirror copied the bad data to both disks, which flipped a bit or two in the file you are using to back the ZFS pool. Since ZFS only see a single device, it has no internal redundancy and can't fix the corrupted bits, only report that it found a block where the on-disk checksum doesn't match the computed checksum of the block. This is why you need to let ZFS handle redundancy via mirror vdevs, raidz vdevs, or (at the very least) copies=2 property on the ZFS filesystem. If there's redundancy in the pool, then ZFS can correct the corruption. > Would it make sense to do "zfs scrub" regularly and have a report sent, > i.e. once a day, so discrepancy would be noticed beforehand? Is there > anything readily available in the Freebsd ZFS package for this? > Without any redundancy in the pool, all a scrub will do is let you know there is corrupted data in the pool. It can't fix it. Neither can gmirror below the pool fix it. All you can do is delete the corrupted file and restore that file from backups. You really should get rid of the gmirror setup, dedicate the entire disks to ZFS, and create a pool using a mirror vdev. File-backed ZFS vdevs really should only be used for testing purposes. -- Freddie Cash fjwc...@gmail.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
Thanks anyone for the help, finally I removed corrupt files from the "current view" of the file system and left the snapshots as they were. This way at least the incremental backup continues. (It is sad that snapshots are so rigid that even corruption is permanent. What more interesting is that, if snapshots are read only, how can they become corrupted?) Would it make sense to do "zfs scrub" regularly and have a report sent, i.e. once a day, so discrepancy would be noticed beforehand? Is there anything readily available in the Freebsd ZFS package for this? B. > From: opensolarisisdeadlongliveopensola...@nedharvey.com > To: sbre...@hotmail.com; zfs-discuss@opensolaris.org > Subject: RE: [zfs-discuss] Remove corrupt files from snapshot > Date: Mon, 14 Nov 2011 19:32:21 -0500 > > > From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > > boun...@opensolaris.org] On Behalf Of sbre...@hotmail.com > > > > Actually a regular file (on a RAID1 setup with gmirror and 2 identical > disks) is > > used as backing store for ZFS. The hardware should be fine as nothing else > > seems to be corrupt. > > In a 10-second google, I see that gmirror is a FreeBSD raid tool, perhaps > similar in some ways to linux lvm. One similarity it has - It doesn't > count. > > You should be using zpool mirroring. Then ZFS will be aware of the > redundant copy, and then ZFS has the potential to correct corruption it > finds. If you're doing the redundancy at a level below ZFS, then ZFS can > only see one device. It cannot perform as well this way, and it cannot > perform such features as redundant copy error correction. > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of sbre...@hotmail.com > > Actually a regular file (on a RAID1 setup with gmirror and 2 identical disks) is > used as backing store for ZFS. The hardware should be fine as nothing else > seems to be corrupt. In a 10-second google, I see that gmirror is a FreeBSD raid tool, perhaps similar in some ways to linux lvm. One similarity it has - It doesn't count. You should be using zpool mirroring. Then ZFS will be aware of the redundant copy, and then ZFS has the potential to correct corruption it finds. If you're doing the redundancy at a level below ZFS, then ZFS can only see one device. It cannot perform as well this way, and it cannot perform such features as redundant copy error correction. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
On Mon, 14 Nov 2011 16:39:25 +, wrote: > >Is ZFS not recommended with file backing store? > >From man zpool: SunOS 5.11 Last change: 24 Nov 2009 2 System Administration Commandszpool(1M) Virtual Devices (vdevs) A "virtual device" describes a single device or a collection of devices organized according to certain performance and fault characteristics. The following virtual devices are supported: disk A block device, typically located under /dev/dsk. ZFS [...] file A regular file. The use of files as a backing store is strongly discouraged. It is designed primarily for experimental purposes, as the fault tolerance of a file is only as good as the file system of which it is a part. A file must be specified by a full path. mirror [] -- ( Kees Nuyt ) c[_] ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
Actually a regular file (on a RAID1 setup with gmirror and 2 identical disks) is used as backing store for ZFS. The hardware should be fine as nothing else seems to be corrupt. Wonder if a server reset could have caused the issue? There are 2 things that surely do not work perfectly: 1. Startup: [root@xxx /etc/rc.d]# cat /etc/rc.conf | grep mdconfig mdconfig_md0="-f /usr/local/zfs/store" [root@xxx /etc/rc.d]# /etc/rc.d/mdconfig start Creating md0 device (-f). mount: /dev/md0: unknown special file or file system I have to run 2 times "zfs mount all", to see the folders. 2. Shutdown: ... +++ /tmp/security.Z3SCbf2M 2011-10-26 03:07:20.0 +0200 +Waiting (max 60 seconds) for system process `vnlru' to stop...done +Waiting (max 60 seconds) for system process `bufdaemon' to stop...done +Wait +iSnygn ci(nmga xd is6k0s ,s evcnoonddess) rfeomra isnyisntge.m. .pr5oc ess `syncer' to stop...4 1 3 3 2 2 0 0 0 done +All buffers synced. The computer does not reboot after this, just waits for ??? . Manual reset is needed. Is ZFS not recommended with file backing store? B. > From: opensolarisisdeadlongliveopensola...@nedharvey.com > To: sbre...@hotmail.com; zfs-discuss@opensolaris.org > Subject: RE: [zfs-discuss] Remove corrupt files from snapshot > Date: Mon, 14 Nov 2011 09:36:58 -0500 > > > From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > > boun...@opensolaris.org] On Behalf Of sbre...@hotmail.com > > > > What does this error mean? I cannot even "scan" the ZFS file system > > anymore? Is there any "fsck" for ZFS? > > There is zpool scrub. It will check all the checksums previously > calculated, verifying the data that was actually written is the data ZFS > previously thought it wrote. If you have sufficient redundancy (mirror or > raid) it will self-correct any errors it finds. > > Since you're experincing corruption that doesn't go away, I'm supposing you > don't have redundancy, or else the corruption happened in something higher > up, such as a failing cpu or non-ecc ram, or a flaky disk controller. > > In any event, do you have any reason to believe you've eliminated the cause > of the corruption? The behavior you're experiencing is normal if you have > failing hardware. > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of sbre...@hotmail.com > > What does this error mean? I cannot even "scan" the ZFS file system > anymore? Is there any "fsck" for ZFS? There is zpool scrub. It will check all the checksums previously calculated, verifying the data that was actually written is the data ZFS previously thought it wrote. If you have sufficient redundancy (mirror or raid) it will self-correct any errors it finds. Since you're experincing corruption that doesn't go away, I'm supposing you don't have redundancy, or else the corruption happened in something higher up, such as a failing cpu or non-ecc ram, or a flaky disk controller. In any event, do you have any reason to believe you've eliminated the cause of the corruption? The behavior you're experiencing is normal if you have failing hardware. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
Back to this topic, since I cannot touch snapshots I thought I could simply
remove the corrupt files after the last snapshot, so the next incremental
backup will notice the difference (i.e. no file) and overwrite the
corrupt-and-removed files with valid ones. This was the plan.
However, while checking for corrupt files, "find" stops at some directory with
"fts_read: Not a directory":
find . -exec md5 {} \; > /home/xxx/md5_out 2> /home/xxx/md5_err &
tail /home/xxx/md5_err
...
md5: ./.zfs/snapshot/20100323081201/Bazsi/Projects/Java Test
Client/java_test_client/lib/xxx/weblogic.jar: Input/output error
md5: ./.zfs/snapshot/20100323081201/@Cache (Bazsi)/BMWi
SP/Publikationen/PDF-Brosch▒ren/Nexxt.pdf: Input/output error
find: fts_read: Not a directory
What does this error mean? I cannot even "scan" the ZFS file system anymore? Is
there any "fsck" for ZFS?
Cheers,
B.
> From: zfsdisc...@orgdotuk.org.uk
> To: zfs-discuss@opensolaris.org
> Date: Mon, 7 Nov 2011 21:49:56 +0000
> Subject: Re: [zfs-discuss] Remove corrupt files from snapshot
>
> > -Original Message-
> > From: Edward Ned Harvey
> > Sent: 04/11/2011 21:23
> >
> > You need to destroy the snapshot completely - But if you want
> > to selectively
> > delete from a snapshot, I think you can clone it, then
> > promote the clone,
> > then destroy the snapshot, then rm something from the clone and then
> > snapshot the clone back to the original name, and then
> > destroy the clone.
> >
> > Right?
>
> Not so fast! :-)
>
> If you promote this new clone, the current state / branch of your filesystem
> becomes a clone instead, dependent on the snapshot.
> Then if you try to destroy the snapshot, you'll fail, because it has a
> dependent clone (your current fs!!!). If you continue
> without realising the implications, and so try the 'destroy' again with '-R',
> there goes the neighbourhood!
>
> I did this once, and was only saved by the fact that my cwd was in my current
> filesystem, so couldn't be unmounted, and therefore
> couldn't be removed! Phew!! Nice to learn something and only get singed
> eyebrows, instead of losing a leg!
>
> hth Andy
>
>
> ___
> zfs-discuss mailing list
> zfs-discuss@opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
> -Original Message- > From: Edward Ned Harvey > Sent: 04/11/2011 21:23 > > You need to destroy the snapshot completely - But if you want > to selectively > delete from a snapshot, I think you can clone it, then > promote the clone, > then destroy the snapshot, then rm something from the clone and then > snapshot the clone back to the original name, and then > destroy the clone. > > Right? Not so fast! :-) If you promote this new clone, the current state / branch of your filesystem becomes a clone instead, dependent on the snapshot. Then if you try to destroy the snapshot, you'll fail, because it has a dependent clone (your current fs!!!). If you continue without realising the implications, and so try the 'destroy' again with '-R', there goes the neighbourhood! I did this once, and was only saved by the fact that my cwd was in my current filesystem, so couldn't be unmounted, and therefore couldn't be removed! Phew!! Nice to learn something and only get singed eyebrows, instead of losing a leg! hth Andy ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of sbre...@hotmail.com > > However, snapshots seem to be read-only: > > Is there any way to force the file removal? You need to destroy the snapshot completely - But if you want to selectively delete from a snapshot, I think you can clone it, then promote the clone, then destroy the snapshot, then rm something from the clone and then snapshot the clone back to the original name, and then destroy the clone. Right? BTW, since snapshots are listed in chronological order, it is distinctly possible the above might cause unintended consequences for snapshot scripts / autosnapshot / whatever. Most people in your situation would simply destroy the snapshot and never look back. That's the easy thing to do. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
On 03 November, 2011 - Paul Kraus sent me these 1,3K bytes:
> On Thu, Nov 3, 2011 at 8:35 AM, wrote:
>
> > I have got a bunch of corrupted files in various snapshots on my
> > ZFS file backing store. I was not able to recover them so decided
> > to remove all, otherwise the continuously make trouble for my
> > incremental backup (rsync, diff etc. fails).
>
> Why are you backing up the snapshots ? Or perhaps a better question is
> why are you backing them up more than once, as they can't change ?
>
> What are you trying to accomplish with the snapshots ?
>
> You can set the snapdir property on the dataset to hidden and it will
> not show up with an ls, even an ls -a, you have to know that the
> ".zfs" directory is there and cd into it blind. This will keep tools
> that walk the directory tree from finding it.
>
> > zfs get snapdir xxx
> NAME PROPERTY VALUESOURCE
> xxx snapdir hidden default
>
> You would use "zfs set snapdir=hidden " to set the parameter.
.. which is default.
/Tomas
--
Tomas Forsman, st...@acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Umeå
`- Sysadmin at {cs,acc}.umu.se
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
On Thu, Nov 3, 2011 at 8:35 AM, wrote:
> I have got a bunch of corrupted files in various snapshots on my
> ZFS file backing store. I was not able to recover them so decided
> to remove all, otherwise the continuously make trouble for my
> incremental backup (rsync, diff etc. fails).
Why are you backing up the snapshots ? Or perhaps a better question is
why are you backing them up more than once, as they can't change ?
What are you trying to accomplish with the snapshots ?
You can set the snapdir property on the dataset to hidden and it will
not show up with an ls, even an ls -a, you have to know that the
".zfs" directory is there and cd into it blind. This will keep tools
that walk the directory tree from finding it.
> zfs get snapdir xxx
NAME PROPERTY VALUESOURCE
xxx snapdir hidden default
You would use "zfs set snapdir=hidden " to set the parameter.
--
{1-2-3-4-5-6-7-}
Paul Kraus
-> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
-> Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
-> Technical Advisor, RPI Players
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Remove corrupt files from snapshot
Hi, snapshots are read-only by design; you can clone them and manipulate the clone, but the snapshot itself remains r/o. HTH Michael On Thu, Nov 3, 2011 at 13:35, wrote: > > Hello, > > I have got a bunch of corrupted files in various snapshots on my ZFS file > backing store. I was not able to recover them so decided to remove all, > otherwise the continuously make trouble for my incremental backup (rsync, > diff etc. fails). > > However, snapshots seem to be read-only: > > # zpool status -v > pool: backups > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: http://www.sun.com/msg/ZFS-8000-8A > scrub: none requested > config: > NAME STATE READ WRITE CKSUM > backups ONLINE 0 0 13 > md0 ONLINE 0 0 13 > errors: Permanent errors have been detected in the following files: > /backups/memory_card/.zfs/snapshot/20110218230726/Backup/Backup.arc > ... > > # rm /backups/memory_card/.zfs/snapshot/20110218230726/Backup/Backup.arc > rm: /backups/memory_card/.zfs/snapshot/20110218230726/Backup/Backup.arc: > Read-only file system > > > Is there any way to force the file removal? > > > Cheers, > B. > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > -- Michael Schuster http://recursiveramblings.wordpress.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Remove corrupt files from snapshot
Hello, I have got a bunch of corrupted files in various snapshots on my ZFS file backing store. I was not able to recover them so decided to remove all, otherwise the continuously make trouble for my incremental backup (rsync, diff etc. fails). However, snapshots seem to be read-only: # zpool status -v pool: backups state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: none requested config: NAMESTATE READ WRITE CKSUM backups ONLINE 0 013 md0 ONLINE 0 013 errors: Permanent errors have been detected in the following files: /backups/memory_card/.zfs/snapshot/20110218230726/Backup/Backup.arc ... # rm /backups/memory_card/.zfs/snapshot/20110218230726/Backup/Backup.arc rm: /backups/memory_card/.zfs/snapshot/20110218230726/Backup/Backup.arc: Read-only file system Is there any way to force the file removal? Cheers, B. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
