Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-25 Thread Robert Milkowski
On 21/01/2010 11:55, Julian Regel wrote: Until you try to pick one up and put it in a fire safe! Then you backup to tape from x4540 whatever data you need. In case of enterprise products you save on licensing here as you need a one client license per x4540 but in fact can backup data from

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-25 Thread Greg
uep, This solution seems like the best and most efficient way of handling large filesystems. My biggest question however is, when backing this up to tape, can it be split across several tapes? I will be using bacula to back this up. Will i need to tar or star this filesystem before writing it

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-22 Thread Mike Gerdts
On Thu, Jan 21, 2010 at 11:28 AM, Richard Elling richard.ell...@gmail.com wrote: On Jan 21, 2010, at 3:55 AM, Julian Regel wrote: Until you try to pick one up and put it in a fire safe! Then you backup to tape from x4540 whatever data you need. In case of enterprise products you save on

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-22 Thread A Darren Dunham
On Wed, Jan 20, 2010 at 08:11:27AM +1300, Ian Collins wrote: True, but I wonder how viable its future is. One of my clients requires 17 LT04 types for a full backup, which cost more and takes up more space than the equivalent in removable hard drives. What kind of removable hard drives are

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-22 Thread A Darren Dunham
On Thu, Jan 21, 2010 at 12:38:56AM +0100, Ragnar Sundblad wrote: On 21 jan 2010, at 00.20, Al Hopper wrote: I remember for about 5 years ago (before LT0-4 days) that streaming tape drives would go to great lengths to ensure that the drive kept streaming - because it took so much time to

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-22 Thread Ian Collins
A Darren Dunham wrote: On Wed, Jan 20, 2010 at 08:11:27AM +1300, Ian Collins wrote: True, but I wonder how viable its future is. One of my clients requires 17 LT04 types for a full backup, which cost more and takes up more space than the equivalent in removable hard drives. What kind

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-21 Thread Robert Milkowski
On 20/01/2010 15:45, David Dyer-Bennet wrote: On Wed, January 20, 2010 09:23, Robert Milkowski wrote: Now you rsync all the data from your clients to a dedicated filesystem per client, then create a snapshot. Is there an rsync out there that can reliably replicate all file

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-21 Thread Robert Milkowski
On 20/01/2010 19:20, Ian Collins wrote: Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-21 Thread Ian Collins
Robert Milkowski wrote: On 20/01/2010 19:20, Ian Collins wrote: Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-21 Thread Andrew Gabriel
Robert Milkowski wrote: I think one should actually compare whole solutions - including servers, fc infrastructure, tape drives, robots, software costs, rack space, ... Servers like x4540 are ideal for zfs+rsync backup solution - very compact, good $/GB ratio, enough CPU power for its

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-21 Thread Robert Milkowski
On 21/01/2010 09:07, Ian Collins wrote: Robert Milkowski wrote: On 20/01/2010 19:20, Ian Collins wrote: Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-21 Thread Julian Regel
Until you try to pick one up and put it in a fire safe! Then you backup to tape from x4540 whatever data you need. In case of enterprise products you save on licensing here as you need a one client license per x4540 but in fact can backup data from many clients which are there. Which brings

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-21 Thread Richard Elling
On Jan 21, 2010, at 3:55 AM, Julian Regel wrote: Until you try to pick one up and put it in a fire safe! Then you backup to tape from x4540 whatever data you need. In case of enterprise products you save on licensing here as you need a one client license per x4540 but in fact can backup

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-21 Thread Ian Collins
Julian Regel wrote: Until you try to pick one up and put it in a fire safe! Then you backup to tape from x4540 whatever data you need. In case of enterprise products you save on licensing here as you need a one client license per x4540 but in fact can backup data from many clients which are

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Ian Collins
Allen Eastwood wrote: On Jan 19, 2010, at 22:54 , Ian Collins wrote: Allen Eastwood wrote: On Jan 19, 2010, at 18:48 , Richard Elling wrote: Many people use send/recv or AVS for disaster recovery on the inexpensive side. Obviously, enterprise backup systems also provide DR

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Ragnar Sundblad
On 19 jan 2010, at 20.11, Ian Collins wrote: Julian Regel wrote: Based on what I've seen in other comments, you might be right. Unfortunately, I don't feel comfortable backing up ZFS filesystems because the tools aren't there to do it (built into the operating system or using

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Joerg Schilling
Richard Elling richard.ell...@gmail.com wrote: ufsdump/restore was perfect in that regard. The lack of equivalent functionality is a big problem for the situations where this functionality is a business requirement. How quickly we forget ufsdump's limitations :-). For example, it

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Joerg Schilling
Ian Collins i...@ianshome.com wrote: The correct way to archivbe ACLs would be to put them into extended POSIX tar attrubutes as star does. See http://cdrecord.berlios.de/private/man/star/star.4.html for a format documentation or have a look at ftp://ftp.berlios.de/pub/star/alpha,

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Joerg Schilling
Edward Ned Harvey sola...@nedharvey.com wrote: Star implements this in a very effective way (by using libfind) that is even faster that the find(1) implementation from Sun. Even if I just find my filesystem, it will run for 7 hours. But zfs can create my whole incremental snapshot in a

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Julian Regel
While I can appreciate that ZFS snapshots are very useful in being able to recover files that users might have deleted, they do not do much to help when the entire disk array experiences a crash/corruption or catches fire. Backing up to a second array helps if a) the array is off-site and for

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Julian Regel
If you like to have a backup that allows to access files, you need a file based backup and I am sure that even a filesystem level scan for recently changed files will not be much faster than what you may achive with e.g. star. Note that ufsdump directly accesees the raw disk device and

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Joerg Schilling
Julian Regel jrmailgate-zfsdisc...@yahoo.co.uk wrote: If you like to have a backup that allows to access files, you need a file based backup and I am sure that even a filesystem level scan for recently changed files will not be much faster than what you may achive with e.g. star.

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Julian Regel
While I am sure that star is technically a fine utility, the problem is that it is effectively an unsupported product. From this viewpoint, you may call most of Solaris unsupported. From the perspective of the business, the contract with Sun provides that support. If our customers find a

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Joerg Schilling
Julian Regel jrmailgate-zfsdisc...@yahoo.co.uk wrote: While I am sure that star is technically a fine utility, the problem is that it is effectively an unsupported product. From this viewpoint, you may call most of Solaris unsupported. From the perspective of the business, the contract

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Michael Schuster
Joerg Schilling wrote: Julian Regel jrmailgate-zfsdisc...@yahoo.co.uk wrote: If you like to have a backup that allows to access files, you need a file based backup and I am sure that even a filesystem level scan for recently changed files will not be much faster than what you may achive with

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Robert Milkowski
On 19/01/2010 19:11, Ian Collins wrote: Julian Regel wrote: Based on what I've seen in other comments, you might be right. Unfortunately, I don't feel comfortable backing up ZFS filesystems because the tools aren't there to do it (built into the operating system or using Zmanda/Amanda).

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Robert Milkowski
On 20/01/2010 10:48, Ragnar Sundblad wrote: On 19 jan 2010, at 20.11, Ian Collins wrote: Julian Regel wrote: Based on what I've seen in other comments, you might be right. Unfortunately, I don't feel comfortable backing up ZFS filesystems because the tools aren't there to do it

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread David Dyer-Bennet
On Wed, January 20, 2010 09:23, Robert Milkowski wrote: Now you rsync all the data from your clients to a dedicated filesystem per client, then create a snapshot. Is there an rsync out there that can reliably replicate all file characteristics between two ZFS/Solaris systems? I haven't

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread David Dyer-Bennet
On Wed, January 20, 2010 04:48, Ragnar Sundblad wrote: LTO media is still cheaper than equivalent sized disks, maybe a factor 5 or so. LTO drivers cost a little, but so do disk shelves. So, now that there is no big price issue, there is choice instead. Use it! Depends on the scale you're

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Bob Friesenhahn
On Wed, 20 Jan 2010, Julian Regel wrote: If our customers find a bug in their backup that is caused by a failure in a Sun supplied utility, then they have a legal course of action. The customer's system administrators are covered because they were using tools provided by the vendor. The

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Julian Regel
It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single pool. This would provide well over 30TB of logical storage per each

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Miles Nordin
ae == Allen Eastwood mi...@paconet.us writes: ic == Ian Collins i...@ianshome.com writes: If people are really still backing up to tapes or DVD's, just use file vdev's, export the pool, and then copy the unmounted vdev onto the tape or DVD. ae And some of those

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Robert Milkowski
On 20/01/2010 16:22, Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single pool. This would provide

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Robert Milkowski
On 20/01/2010 17:21, Robert Milkowski wrote: On 20/01/2010 16:22, Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Richard Elling
On Jan 20, 2010, at 3:15 AM, Joerg Schilling wrote: Richard Elling richard.ell...@gmail.com wrote: ufsdump/restore was perfect in that regard. The lack of equivalent functionality is a big problem for the situations where this functionality is a business requirement. How quickly we

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Ian Collins
Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single pool. This would provide well over 30TB of

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Ian Collins
Joerg Schilling wrote: Ian Collins i...@ianshome.com wrote: The correct way to archivbe ACLs would be to put them into extended POSIX tar attrubutes as star does. See http://cdrecord.berlios.de/private/man/star/star.4.html for a format documentation or have a look at

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Miles Nordin
jr == Julian Regel jrmailgate-zfsdisc...@yahoo.co.uk writes: jr While I am sure that star is technically a fine utility, the jr problem is that it is effectively an unsupported product. I have no problems with this whatsoever. jr If our customers find a bug in their backup that is

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread David Magda
On Jan 20, 2010, at 12:21, Robert Milkowski wrote: On 20/01/2010 16:22, Julian Regel wrote: [...] So you could provision a tape backup for just under £3 (~ $49000). In comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price £30900. I've not factored in backup

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Joerg Schilling
Ian Collins i...@ianshome.com wrote: We are talking about TAR and I did give a pointer to the star archive format documentation, so it is obvious that I was talking about the ACL format from Sun tar. This format is not documented. It is, Sun's ZFS ACL aware tools use acltotext()

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Joerg Schilling
Miles Nordin car...@ivy.net wrote: From the perspective of MY business, I would much rather have the dark OOB acl/fork/whatever-magic that's gone into ZFS and NFSv4 supported in standard tools like rsync and GNUtar. This is, for example, what GNU tar does not support any platform speficic

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Al Hopper
On Wed, Jan 20, 2010 at 2:52 PM, David Magda dma...@ee.ryerson.ca wrote: On Jan 20, 2010, at 12:21, Robert Milkowski wrote: On 20/01/2010 16:22, Julian Regel wrote: [...] So you could provision a tape backup for just under £3 (~$49000). In comparison, the cost of one X4540 with ~ 36TB

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Ragnar Sundblad
On 20 jan 2010, at 17.22, Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single pool. This

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Ragnar Sundblad
On 21 jan 2010, at 00.20, Al Hopper wrote: I remember for about 5 years ago (before LT0-4 days) that streaming tape drives would go to great lengths to ensure that the drive kept streaming - because it took so much time to stop, backup and stream again. And one way the drive firmware

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Joerg Schilling
Ragnar Sundblad ra...@csc.kth.se wrote: Yes! Modern LTO drives can typically vary their speed about a factor four or so, so even if you can't keep up with the tape drive maximum speed, it will typically work pretty good anyway. If you can't keep up even then, it will have to stop, back up a

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Joerg Schilling
Thomas Burgess wonsl...@gmail.com wrote: so star is better than tar? Star is the oldest OSS tar implementation (it started in 1982). Star is (in contrary to Sun's tar and to GNU tar) able to create archives with attributes from recent POSIX standards and it implements aprox. twice as many

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Joerg Schilling
Edward Ned Harvey sola...@nedharvey.com wrote: I still believe that a set of compressed incremental star archives give you more features. Big difference there is that in order to create an incremental star archive, star has to walk the whole filesystem or folder that's getting backed up,

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Joerg Schilling
Lassi Tuura l...@cern.ch wrote: I guess what I am after is, for data which really matters to its owners and which they actually had to recover, did people use tar/pax archives (~ file level standard archive format), dump/restore (~ semi-standard format based on files/inodes) or zfs

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Joerg Schilling
Miles Nordin car...@ivy.net wrote: When we brought it up last time, I think we found no one knows of a userland tool similar to 'ufsdump' that's capable of serializing a ZFS along with holes, large files, ``attribute'' forks, windows ACL's, and checksums of its own, and then restoring the

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Joerg Schilling
Richard Elling richard.ell...@gmail.com wrote: OOB, the default OpenSolaris PATH places /usr/gnu/bin ahead of /usr/bin, so gnu tar is the default. As of b130 (I'm not running an older build currently) the included gnu tar is version 1.22 which is the latest as released March 2009 at

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Joerg Schilling
Daniel Carosone d...@geek.com.au wrote: I also don't recommend files 1Gb in size for DVD media, due to iso9660 limitations. I haven't used UDF enough to say much about any limitations there. ISO-9660 supports files up to 8 TB. Do you have a bigger pool? Jörg --

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Jerry K
+1 for zfsdump/zfsrestore Julian Regel wrote: When we brought it up last time, I think we found no one knows of a userland tool similar to 'ufsdump' that's capable of serializing a ZFS along with holes, large files, ``attribute'' forks, windows ACL's, and checksums of its own, and then

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Richard Elling
On Jan 19, 2010, at 1:53 AM, Julian Regel wrote: When we brought it up last time, I think we found no one knows of a userland tool similar to 'ufsdump' that's capable of serializing a ZFS along with holes, large files, ``attribute'' forks, windows ACL's, and checksums of its own, and then

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Tim Cook
On Tue, Jan 19, 2010 at 11:36 AM, Richard Elling richard.ell...@gmail.comwrote: On Jan 19, 2010, at 1:53 AM, Julian Regel wrote: When we brought it up last time, I think we found no one knows of a userland tool similar to 'ufsdump' that's capable of serializing a ZFS along with holes,

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Julian Regel
The beauty of ufsdump/ufsrestore is that because it's bundled with the operating system, I can perform bare metal recovery using a Solaris DVD and locally attached tape drive. It's simple and arguably essential for system administrators. Yep. And it was invented because there was no

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Ian Collins
Julian Regel wrote: Based on what I've seen in other comments, you might be right. Unfortunately, I don't feel comfortable backing up ZFS filesystems because the tools aren't there to do it (built into the operating system or using Zmanda/Amanda). Commercial backup solutions are available

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Joerg Schilling
Julian Regel jrmailgate-zfsdisc...@yahoo.co.uk wrote: When I look at the documentation for Zmanda (http://docs.zmanda.com/Project:Amanda_Enterprise_3.0/ZMC_Users_Manual/Appendix_A:_Backing_Up_and_Restoring_Solaris_ZFS), it states that the command used to backup a ZFS filesystem depends on

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Ian Collins
Joerg Schilling wrote: Ian Collins i...@ianshome.com wrote: Julian Regel wrote: Based on what I've seen in other comments, you might be right. Unfortunately, I don't feel comfortable backing up ZFS filesystems because the tools aren't there to do it (built into the operating system

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Bob Friesenhahn
On Wed, 20 Jan 2010, Ian Collins wrote: Commercial backup solutions are available for ZFS. I know tape backup isn't sexy, but it's a reality for many of us and it's not going away anytime soon. True, but I wonder how viable its future is. One of my clients requires 17 LT04 types for a full

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Joerg Schilling
Ian Collins i...@ianshome.com wrote: Sun's tar also writes ACLs in a way that is 100% non-portable. Star cannot understand them and probably never will be able to understand this format as it is not well defined for a portable program like star. Is that because they are NFSv4

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Miles Nordin
jk == Jerry K sun.mail.lis...@oryx.cc writes: jk +1 jk for zfsdump/zfsrestore -1 I don't think a replacement for the ufsdump/ufsrestore tool is really needed. From now on, backups just go into Another Zpool. We need the zfs send stream format commitment (stream format depends only

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Miles Nordin
ic == Ian Collins i...@ianshome.com writes: I know tape backup isn't sexy, but it's a reality for many of us and it's not going away anytime soon. ic True, but I wonder how viable its future is. One of my ic clients requires 17 LT04 types for a full backup, which cost

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Daniel Carosone
On Tue, Jan 19, 2010 at 12:16:01PM +0100, Joerg Schilling wrote: Daniel Carosone d...@geek.com.au wrote: I also don't recommend files 1Gb in size for DVD media, due to iso9660 limitations. I haven't used UDF enough to say much about any limitations there. ISO-9660 supports files up to

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Daniel Carosone
There is a tendency to conflate backup and archive, both generally and in this thread. They have different requirements. Backups should enable quick restore of a full operating image with all the necessary system level attributes. They concerned with SLA and uptime and outage and data loss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Allen Eastwood
Message: 3 Date: Tue, 19 Jan 2010 15:48:52 -0500 From: Miles Nordin car...@ivy.net To: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] zfs send/receive as backup - reliability? Message-ID: oqpr55lt1n@castrovalva.ivy.net Content-Type: text/plain; charset=us-ascii I don't think

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Richard Elling
On Jan 19, 2010, at 4:26 PM, Allen Eastwood wrote: Message: 3 Date: Tue, 19 Jan 2010 15:48:52 -0500 From: Miles Nordin car...@ivy.net To: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] zfs send/receive as backup - reliability? Message-ID: oqpr55lt1n@castrovalva.ivy.net Content

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Allen Eastwood
On Jan 19, 2010, at 18:48 , Richard Elling wrote: Many people use send/recv or AVS for disaster recovery on the inexpensive side. Obviously, enterprise backup systems also provide DR capabilities. Since ZFS has snapshots that actually work, and you can use send/receive or other backup

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Mike La Spina
I use zfs send/recv in the enterprise and in smaller environments all time and it's is excellent. Have a look at how awesome the functionally is in this example. http://blog.laspina.ca/ubiquitous/provisioning_disaster_recovery_with_zfs Regards, Mike -- This message posted from

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Ian Collins
Joerg Schilling wrote: Ian Collins i...@ianshome.com wrote: Sun's tar also writes ACLs in a way that is 100% non-portable. Star cannot understand them and probably never will be able to understand this format as it is not well defined for a portable program like star. Is that

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Ian Collins
Allen Eastwood wrote: On Jan 19, 2010, at 18:48 , Richard Elling wrote: Many people use send/recv or AVS for disaster recovery on the inexpensive side. Obviously, enterprise backup systems also provide DR capabilities. Since ZFS has snapshots that actually work, and you can use send/receive

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Edward Ned Harvey
Star implements this in a very effective way (by using libfind) that is even faster that the find(1) implementation from Sun. Even if I just find my filesystem, it will run for 7 hours. But zfs can create my whole incremental snapshot in a minute or two. There is no way star or any other

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Allen Eastwood
On Jan 19, 2010, at 22:54 , Ian Collins wrote: Allen Eastwood wrote: On Jan 19, 2010, at 18:48 , Richard Elling wrote: Many people use send/recv or AVS for disaster recovery on the inexpensive side. Obviously, enterprise backup systems also provide DR capabilities. Since ZFS has

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-18 Thread Edward Ned Harvey
I still believe that a set of compressed incremental star archives give you more features. Big difference there is that in order to create an incremental star archive, star has to walk the whole filesystem or folder that's getting backed up, and do a stat on every file to see which files have

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-18 Thread Edward Ned Harvey
Consider then, using a zpool-in-a-file as the file format, rather than zfs send streams. That's a pretty cool idea. Then you've still got the entire zfs volume inside of a file, but you're able to mount and extract individual files if you want, and you're able to pipe your zfs send directly to

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-18 Thread Phil Harman
YMMV. At a recent LOSUG meeting we were told of a case where rsync was faster than an incremental zfs send/recv. But I think that was for a mail server with many tiny files (i.e. changed blocks are very easy to find in files with very few blocks). However, I don't see why further ZFS

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-18 Thread Thomas Burgess
On Mon, Jan 18, 2010 at 3:59 AM, Phil Harman phil.har...@gmail.com wrote: YMMV. At a recent LOSUG meeting we were told of a case where rsync was faster than an incremental zfs send/recv. But I think that was for a mail server with many tiny files (i.e. changed blocks are very easy to find in

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-18 Thread Robert Milkowski
On 18/01/2010 08:59, Phil Harman wrote: YMMV. At a recent LOSUG meeting we were told of a case where rsync was faster than an incremental zfs send/recv. But I think that was for a mail server with many tiny files (i.e. changed blocks are very easy to find in files with very few blocks).

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-18 Thread Robert Milkowski
or you might do something like: http://milek.blogspot.com/2009/12/my-presentation-at-losug.html however in your case if all your clients are running zfs only filesystems then relaying just on zfs send|recv might be a good idea. -- Robert Milkowski http://milek.blogspot.com

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-18 Thread Lassi Tuura
Hi, .. it's hard to beat the convenience of a backup file format, for all sorts of reasons, including media handling, integration with other services, and network convenience. Yes. Consider then, using a zpool-in-a-file as the file format, rather than zfs send streams. This is an

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-18 Thread Miles Nordin
mg == Mike Gerdts mger...@gmail.com writes: tt == Toby Thain t...@telegraphics.com.au writes: tb == Thomas Burgess wonsl...@gmail.com writes: mg Yet it is used in ZFS flash archives on Solaris 10 and are mg slated for use in the successor to flash archives. in FLAR, ``if a single

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-18 Thread Richard Elling
On Jan 18, 2010, at 11:04 AM, Miles Nordin wrote: ... Another problem is that the snv_112 man page says this: -8- The format of the stream is evolving. No backwards com- patibility is guaranteed. You may not be able to receive your streams on future versions

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-18 Thread Daniel Carosone
On Mon, Jan 18, 2010 at 07:34:51PM +0100, Lassi Tuura wrote: Consider then, using a zpool-in-a-file as the file format, rather than zfs send streams. This is an interesting suggestion :-) Did I understand you correctly that once a slice is written, zfs won't rewrite it? In other words,

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-18 Thread Robert Milkowski
On 18/01/2010 18:28, Lassi Tuura wrote: Hi, Here is the big difference. For a professional backup people still typically use tapes although tapes have become expensive. I still believe that a set of compressed incremental star archives give you more features. Thanks for your

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-18 Thread Daniel Carosone
On Mon, Jan 18, 2010 at 01:38:16PM -0800, Richard Elling wrote: The Solaris 10 10/09 zfs(1m) man page says: The format of the stream is committed. You will be able to receive your streams on future versions of ZFS. I'm not sure when that hit snv, but obviously it was

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-17 Thread Edward Ned Harvey
NO, zfs send is not a backup. Understood, but perhaps you didn't read my whole message. Here, I will spell out the whole discussion: If you zfs send somefile it is well understood there are two big problems with this method of backup. #1 If a single bit error is introduced into the file,

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-17 Thread Joerg Schilling
Toby Thain t...@telegraphics.com.au wrote: Yet it is used in ZFS flash archives on Solaris 10 I can see the temptation, but isn't it a bit under-designed? I think Mr Nordin might have ranted about this in the past... Isn't flash cpio based and thus not prepared for the future? Cpio coes

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-17 Thread Joerg Schilling
Edward Ned Harvey sola...@nedharvey.com wrote: NO, zfs send is not a backup. Understood, but perhaps you didn't read my whole message. Here, I will spell out the whole discussion: ... Instead, it is far preferable to zfs send | zfs receive ... That is, receive the data stream on

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-17 Thread Thomas Burgess
Cpio coes not support sparse files and is unable to archive files 8 GB. Jörg I found this out the hard way last time i used it. I was backing up all my data from one system to another using cpio and i had a bunch of movies over 8GB (720p and 1080p mkv files) none of them worked. I

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-17 Thread Daniel Carosone
On Sun, Jan 17, 2010 at 05:31:39AM -0500, Edward Ned Harvey wrote: Instead, it is far preferable to zfs send | zfs receive ... That is, receive the data stream on external media as soon as you send it. Agree 100% - but.. .. it's hard to beat the convenience of a backup file format, for all

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-17 Thread Thomas Burgess
What dou you use instead? * * **tar cvf - /some/dir | (cd /some/other/dir; tar xf -) BTW: I recommend star and to use either the H=exustar or -dump option. Jörg i will have to check it out. I recently migrated to opensolaris from FreeBSD and i have a LOT to learn. I am really

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-16 Thread Edward Ned Harvey
I am considering building a modest sized storage system with zfs. Some of the data on this is quite valuable, some small subset to be backed up forever, and I am evaluating back-up options with that in mind. You don't need to store the zfs send data stream on your backup media. This would be

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-16 Thread Joerg Schilling
Edward Ned Harvey sola...@nedharvey.com wrote: I am considering building a modest sized storage system with zfs. Some of the data on this is quite valuable, some small subset to be backed up forever, and I am evaluating back-up options with that in mind. You don't need to store the zfs

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-16 Thread Thomas Burgess
NO, zfs send is not a backup. From a backup, you could restore individual files. Jörg I disagree. It is a backup. It's just not an enterprise backup solution ___ zfs-discuss mailing list zfs-discuss@opensolaris.org

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-16 Thread Toby Thain
On 16-Jan-10, at 7:30 AM, Edward Ned Harvey wrote: I am considering building a modest sized storage system with zfs. Some of the data on this is quite valuable, some small subset to be backed up forever, and I am evaluating back-up options with that in mind. You don't need to store the zfs

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-16 Thread Mike Gerdts
On Sat, Jan 16, 2010 at 5:31 PM, Toby Thain t...@telegraphics.com.au wrote: On 16-Jan-10, at 7:30 AM, Edward Ned Harvey wrote: I am considering building a modest sized storage system with zfs. Some of the data on this is quite valuable, some small subset to be backed up forever, and I am

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-16 Thread Toby Thain
On 16-Jan-10, at 6:51 PM, Mike Gerdts wrote: On Sat, Jan 16, 2010 at 5:31 PM, Toby Thain t...@telegraphics.com.au wrote: On 16-Jan-10, at 7:30 AM, Edward Ned Harvey wrote: I am considering building a modest sized storage system with zfs. Some of the data on this is quite valuable, some

[zfs-discuss] zfs send/receive as backup - reliability?

2010-01-15 Thread Lassi Tuura
Hi, I am considering building a modest sized storage system with zfs. Some of the data on this is quite valuable, some small subset to be backed up forever, and I am evaluating back-up options with that in mind. My understanding is that zfs send approximately captures the copy-on-write file