Re: [zfs-discuss] check a zfs rcvd file
Lori Alt wrote: On 09/04/09 10:17, dick hoogendijk wrote: Lori Alt wrote: The -u option to zfs recv (which was just added to support flash archive installs, but it's useful for other reasons too) suppresses all mounts of the received file systems. So you can mount them yourself afterward in whatever order is appropriate, or do a 'zfs mount -a'. You misunderstood my problem. It is very convenient that the filesystems are not mounted. I only wish they could stay that way!. Alas, they ARE mounted (even if I don't want them to) when I *reboot* the system. And THAT's when thing get ugly. I then have different zfs filesystems using the same mountpoints! The backed up ones have the same mountpoints as their origin :-/ -> The only way to stop it is to *export* the "backup" zpool OR to change *manualy* the zfs prop "canmount=noauto" in all backed up snapshots/filesystems. As I understand I cannot give this "canmount=noauto" to the zfs receive command. # zfs send -Rv rp...@0909 | zfs receive -Fdu backup/snaps There is a RFE to allow zfs recv to assign properties, but I'm not sure whether it would help in your case. I would have thought that "canmount=noauto" would have already been set on the sending side, however. In that case, the property should be preserved when the stream is preserved. Well, I checked again today. This is what happens: NAME PROPERTY VALUESOURCE tank/ROOT/daffy canmount on default NAME PROPERTY VALUE SOURCE rpool/ROOT/daffy canmount noautolocal As you can see the original dataset (rpool/ROOT/daffy) has canmount=noauto set. However, the received dataset (zfs send rpool/ROOT/da...@090905 | zfs receive -Fdu /tank) has this property changed(!) into canmount=on. So, what you state is not true. The property is NOT preserved. Is this a bug? -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS 10u7 5/09 | OpenSolaris 2010.02 B121 + All that's really worth doing is what we do for others (Lewis Carrol) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] check a zfs rcvd file
On 09/04/09 10:17, dick hoogendijk wrote: Lori Alt wrote: The -u option to zfs recv (which was just added to support flash archive installs, but it's useful for other reasons too) suppresses all mounts of the received file systems. So you can mount them yourself afterward in whatever order is appropriate, or do a 'zfs mount -a'. You misunderstood my problem. It is very convenient that the filesystems are not mounted. I only wish they could stay that way!. Alas, they ARE mounted (even if I don't want them to) when I *reboot* the system. And THAT's when thing get ugly. I then have different zfs filesystems using the same mountpoints! The backed up ones have the same mountpoints as their origin :-/ -> The only way to stop it is to *export* the "backup" zpool OR to change *manualy* the zfs prop "canmount=noauto" in all backed up snapshots/filesystems. As I understand I cannot give this "canmount=noauto" to the zfs receive command. # zfs send -Rv rp...@0909 | zfs receive -Fdu backup/snaps There is a RFE to allow zfs recv to assign properties, but I'm not sure whether it would help in your case. I would have thought that "canmount=noauto" would have already been set on the sending side, however. In that case, the property should be preserved when the stream is preserved. But if for some reason, you're not setting that property on the sending side, but want it set on the receiving side, you might have to write a script to set the properties for all those datasets after they are received. lori ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] check a zfs rcvd file
Lori Alt wrote: The -u option to zfs recv (which was just added to support flash archive installs, but it's useful for other reasons too) suppresses all mounts of the received file systems. So you can mount them yourself afterward in whatever order is appropriate, or do a 'zfs mount -a'. You misunderstood my problem. It is very convenient that the filesystems are not mounted. I only wish they could stay that way!. Alas, they ARE mounted (even if I don't want them to) when I *reboot* the system. And THAT's when thing get ugly. I then have different zfs filesystems using the same mountpoints! The backed up ones have the same mountpoints as their origin :-/ -> The only way to stop it is to *export* the "backup" zpool OR to change *manualy* the zfs prop "canmount=noauto" in all backed up snapshots/filesystems. As I understand I cannot give this "canmount=noauto" to the zfs receive command. # zfs send -Rv rp...@0909 | zfs receive -Fdu backup/snaps -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS 10u7 5/09 | OpenSolaris 2010.02 B121 + All that's really worth doing is what we do for others (Lewis Carrol) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] check a zfs rcvd file
On 09/04/09 09:41, dick hoogendijk wrote: Lori Alt wrote: The -n option does some verification. It verifies that the record headers distributed throughout the stream are syntactically valid. Since each record header contains a length field which allows the next header to be found, one bad header will cause the processing of the stream to abort. But it doesn't verify the content of the data associated with each record. So, storing the stream in a zfs received filesystem is the better option. Alas, it also is the most difficult one. Storing to a file with "zfs send -Rv" is easy. The result is just a file and if your reboot the system all is OK. However, if I zfs "receive -Fdu" into a zfs filesystem I'm in trouble when I reboot the system. I get confusion on mountpoints! Let me explain: Some time ago I backup up my rpool and my /export ; /export/home to /backup/snaps (with zfs receive -Fdu). All's OK because the newly created zfs FS's stay unmounted 'till the next reboot(!). When I rebooted my system (due to a kernel upgrade) the system would nog boot, because it had mounted the zfs FS "backup/snaps/export" on /export and "backup/snaps/export/home on /export/home. The system itself had those FS's too, of course. So, there was a mix up. It would be nice if the backup FS's would not be mounted (canmount=noauto), but I cannot give this option when I create the zfs send | receive, can I? And giving this option later on is very difficult, because "canmount" is NOT recursive! And I don't want to set it manualy on all those backup up FS's. I wonder how other people overcome this mountpoint issue. The -u option to zfs recv (which was just added to support flash archive installs, but it's useful for other reasons too) suppresses all mounts of the received file systems. So you can mount them yourself afterward in whatever order is appropriate, or do a 'zfs mount -a'. lori ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] check a zfs rcvd file
Lori Alt wrote: The -n option does some verification. It verifies that the record headers distributed throughout the stream are syntactically valid. Since each record header contains a length field which allows the next header to be found, one bad header will cause the processing of the stream to abort. But it doesn't verify the content of the data associated with each record. So, storing the stream in a zfs received filesystem is the better option. Alas, it also is the most difficult one. Storing to a file with "zfs send -Rv" is easy. The result is just a file and if your reboot the system all is OK. However, if I zfs "receive -Fdu" into a zfs filesystem I'm in trouble when I reboot the system. I get confusion on mountpoints! Let me explain: Some time ago I backup up my rpool and my /export ; /export/home to /backup/snaps (with zfs receive -Fdu). All's OK because the newly created zfs FS's stay unmounted 'till the next reboot(!). When I rebooted my system (due to a kernel upgrade) the system would nog boot, because it had mounted the zfs FS "backup/snaps/export" on /export and "backup/snaps/export/home on /export/home. The system itself had those FS's too, of course. So, there was a mix up. It would be nice if the backup FS's would not be mounted (canmount=noauto), but I cannot give this option when I create the zfs send | receive, can I? And giving this option later on is very difficult, because "canmount" is NOT recursive! And I don't want to set it manualy on all those backup up FS's. I wonder how other people overcome this mountpoint issue. -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS 10u7 5/09 | OpenSolaris 2010.02 b122 + All that's really worth doing is what we do for others (Lewis Carrol) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] check a zfs rcvd file
On 09/03/09 15:25, dick hoogendijk wrote: On Thu, 03 Sep 2009 14:32:27 -0600 Lori Alt wrote: I'm looking at integrating a utility which allows the metadata in a stream to be dumped for debugging purposes (zstreamdump). It also verifies that the data in the stream agrees with the checksum. This sounds nice. Any change this will make it into U8? No, it won't be in U8. lori ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] check a zfs rcvd file
On Thu, 03 Sep 2009 14:32:27 -0600 Lori Alt wrote: > I'm looking at integrating a utility which allows the metadata in a > stream to be dumped for debugging purposes (zstreamdump). It also > verifies that the data in the stream agrees with the checksum. This sounds nice. Any change this will make it into U8? -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS 10u7 5/09 | OpenSolaris 2010.02 B121 + All that's really worth doing is what we do for others (Lewis Carrol) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] check a zfs rcvd file
On 09/03/09 14:21, dick hoogendijk wrote: On Wed, 2 Sep 2009 13:06:35 -0500 (CDT) Bob Friesenhahn wrote: Nothing prevents validating the self-verifying archive file via this "zfs recv -vn " technique. Does this verify the ZFS format/integrity of the stream? Or is the only way to do that to zfs recv the stream into ZFS? The -n option does some verification. It verifies that the record headers distributed throughout the stream are syntactically valid. Since each record header contains a length field which allows the next header to be found, one bad header will cause the processing of the stream to abort. But it doesn't verify the content of the data associated with each record. We might want to implement an option to enhance zfs recv -n to calculate a checksum of each dataset's records as it's reading the stream and then verify the checksum when the dataset's "END" record is seen. I'm looking at integrating a utility which allows the metadata in a stream to be dumped for debugging purposes (zstreamdump). It also verifies that the data in the stream agrees with the checksum. lori ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] check a zfs rcvd file
On Wed, 2 Sep 2009 13:06:35 -0500 (CDT) Bob Friesenhahn wrote: > Nothing prevents validating the self-verifying archive file via this > "zfs recv -vn " technique. Does this verify the ZFS format/integrity of the stream? Or is the only way to do that to zfs recv the stream into ZFS? -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS 10u7 5/09 | OpenSolaris 2010.02 B121 + All that's really worth doing is what we do for others (Lewis Carrol) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] check a zfs rcvd file
Dick Hoogendijk wrote: Some time ago there was some discussion on zfs send | rcvd TO A FILE. Apart form the disadvantages which I now know someone mentioned a CHECK to be at least sure that the file itself was OK (without one or more bits that felt over). I lost this reply and would love to hear this check again. In other words how can I be sure of the validity of the received file in the next command line: # zfs send -Rv rp...@090902 > /backup/snaps/rpool.090902 I only want to know how to check the integrity of the received file. You should be able to generate a sha1sum/md5sum of the zfs send stream on the fly with 'tee': # zfs send -R rp...@090902 | tee /backups/snaps/rpool.090902 | sha1sum compare the output of that with the sha1sum of the file on-disk: # sha1sum /backups/snaps/rpool.090902 This only guarantees that the file contains the exact same bits as the zfs send stream. It does not verify the ZFS format/integrity of the stream - the only way to do that is to zfs recv the stream into ZFS. -- Dave ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] check a zfs rcvd file
On Wed, 2 Sep 2009, Will Murnane wrote: "zfs recv -vn < file" will check the integrity of the zfs stream in file. However, this is only a one-time check; if the data is corrupted later the stream will not be recoverable. You might consider using something like par2 [1] to generate parity: The most commonly available program which self-validates data is 'gzip' but 'lzop' is faster. Besides self-validation, there is the advantage that the data is smaller due to compression. Nothing prevents validating the self-verifying archive file via this "zfs recv -vn " technique. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] check a zfs rcvd file
"zfs recv -vn < file" will check the integrity of the zfs stream in file. However, this is only a one-time check; if the data is corrupted later the stream will not be recoverable. You might consider using something like par2 [1] to generate parity: while true: zfs send f...@snap > file generate par2 at desired strength if zfs recv -vn < file succeeds: break The reason for generating parity before checking if the recv succeeds is then you can be reasonably sure the stream that the parity was generated for is a valid stream. Then store the stream and the parity files together, and use the parity files to recover from media damage over time. Will [1]: http://parchive.sourceforge.net/ On Wed, Sep 2, 2009 at 13:31, Dick Hoogendijk wrote: > > Some time ago there was some discussion on zfs send | rcvd TO A FILE. > Apart form the disadvantages which I now know someone mentioned a CHECK to > be at least sure that the file itself was OK (without one or more bits that > felt over). I lost this reply and would love to hear this check again. In > other words how can I be sure of the validity of the received file in the > next command line: > > # zfs send -Rv rp...@090902 > /backup/snaps/rpool.090902 > > I only want to know how to check the integrity of the received file. > > -- > Dick Hoogendijk -- PGP/GnuPG key: 01D2433D > + http://nagual.nl/ | SunOS 10u7 5/09 | OpenSolaris 2010.02 B121 > + All that's really worth doing is what we do for others (Lewis Carrol) > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] check a zfs rcvd file
Some time ago there was some discussion on zfs send | rcvd TO A FILE. Apart form the disadvantages which I now know someone mentioned a CHECK to be at least sure that the file itself was OK (without one or more bits that felt over). I lost this reply and would love to hear this check again. In other words how can I be sure of the validity of the received file in the next command line: # zfs send -Rv rp...@090902 > /backup/snaps/rpool.090902 I only want to know how to check the integrity of the received file. -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS 10u7 5/09 | OpenSolaris 2010.02 B121 + All that's really worth doing is what we do for others (Lewis Carrol) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
