Re: [zfs-discuss] ZFS Distro Advice

2013-03-05 Thread Robert Milkowski
We do the same for all of our legacy operating system backups. Take a snapshot then do an rsync and an excellent way of maintaining incremental backups for those. Magic rsync options used: -a --inplace --no-whole-file --delete-excluded This causes rsync to overwrite the file

Re: [zfs-discuss] ZFS Distro Advice

2013-03-05 Thread Bob Friesenhahn
On Mon, 4 Mar 2013, Matthew Ahrens wrote: Magic rsync options used: -a --inplace --no-whole-file --delete-excluded This causes rsync to overwrite the file blocks in place rather than writing to a new temporary file first. As a result, zfs COW produces primitive deduplication of at least

Re: [zfs-discuss] ZFS Distro Advice

2013-03-05 Thread David Magda
On Tue, March 5, 2013 10:02, Bob Friesenhahn wrote: Rsync does need to read files on the destination filesystem to see if they have changed. If the system has sufficient RAM (and/or L2ARC) then files may still be cached from the previous day's run. In most cases only a small subset of the

Re: [zfs-discuss] ZFS Distro Advice

2013-03-05 Thread Russ Poyner
On 3/5/2013 9:40 AM, David Magda wrote: On Tue, March 5, 2013 10:02, Bob Friesenhahn wrote: Rsync does need to read files on the destination filesystem to see if they have changed. If the system has sufficient RAM (and/or L2ARC) then files may still be cached from the previous day's run. In

Re: [zfs-discuss] ZFS Distro Advice

2013-03-05 Thread Bob Friesenhahn
On Tue, 5 Mar 2013, David Magda wrote: It's also possible to reduce the amount that rsync has to walk the entire file tree. Most folks simply do a rsync --options /my/source/ /the/dest/, but if you use zfs diff, and parse/feed the output of that to rsync, then the amount of thrashing can

Re: [zfs-discuss] ZFS Distro Advice

2013-03-05 Thread Russ Poyner
On 3/5/2013 10:27 AM, Bob Friesenhahn wrote: On Tue, 5 Mar 2013, David Magda wrote: It's also possible to reduce the amount that rsync has to walk the entire file tree. Most folks simply do a rsync --options /my/source/ /the/dest/, but if you use zfs diff, and parse/feed the output of that to

Re: [zfs-discuss] ZFS Distro Advice

2013-03-05 Thread David Magda
On Tue, March 5, 2013 11:17, Russ Poyner wrote: Your idea to use zfs diff to limit the need to stat the entire filesystem tree intrigues me. My current rsync backups are normally limited by this very factor. It takes longer to walk the filesystem tree than it does to transfer the new data.

Re: [zfs-discuss] ZFS Distro Advice

2013-03-04 Thread Matthew Ahrens
On Tue, Feb 26, 2013 at 7:42 PM, Bob Friesenhahn bfrie...@simple.dallas.tx.us wrote: On Wed, 27 Feb 2013, Ian Collins wrote: I am finding that rsync with the right options (to directly block-overwrite) plus zfs snapshots is providing me with pretty amazing deduplication for backups without

Re: [zfs-discuss] ZFS Distro Advice

2013-02-27 Thread Ahmed Kamal
How is the quality of the ZFS Linux port today? Is it comparable to Illumos or at least FreeBSD ? Can I trust production data to it ? On Wed, Feb 27, 2013 at 5:22 AM, Bob Friesenhahn bfrie...@simple.dallas.tx.us wrote: On Tue, 26 Feb 2013, Gary Driggs wrote: On Feb 26, 2013, at 12:44 AM,

Re: [zfs-discuss] ZFS Distro Advice

2013-02-27 Thread Sašo Kiselkov
On 02/27/2013 12:32 PM, Ahmed Kamal wrote: How is the quality of the ZFS Linux port today? Is it comparable to Illumos or at least FreeBSD ? Can I trust production data to it ? Can't speak from personal experience, but a colleague of mine has been PPA builds on Ubuntu and has had, well, less

Re: [zfs-discuss] ZFS Distro Advice

2013-02-27 Thread Dan Swartzendruber
:37 AM To: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] ZFS Distro Advice On 02/27/2013 12:32 PM, Ahmed Kamal wrote: How is the quality of the ZFS Linux port today? Is it comparable to Illumos or at least FreeBSD ? Can I trust production data to it ? Can't speak from personal experience

Re: [zfs-discuss] ZFS Distro Advice

2013-02-27 Thread Bob Friesenhahn
On Wed, 27 Feb 2013, Ian Collins wrote: Magic rsync options used: -a --inplace --no-whole-file --delete-excluded This causes rsync to overwrite the file blocks in place rather than writing to a new temporary file first. As a result, zfs COW produces primitive deduplication of at least the

Re: [zfs-discuss] ZFS Distro Advice

2013-02-27 Thread Tim Cook
On Wed, Feb 27, 2013 at 2:57 AM, Dan Swartzendruber dswa...@druber.comwrote: I've been using it since rc13. It's been stable for me as long as you don't get into things like zvols and such... Then it definitely isn't at the level of FreeBSD, and personally I would not consider that

Re: [zfs-discuss] ZFS Distro Advice

2013-02-27 Thread Dan Swartzendruber
On 2/27/2013 2:05 PM, Tim Cook wrote: On Wed, Feb 27, 2013 at 2:57 AM, Dan Swartzendruber dswa...@druber.com mailto:dswa...@druber.com wrote: I've been using it since rc13. It's been stable for me as long as you don't get into things like zvols and such... Then it

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Tiernan OToole
Thanks all! I will check out FreeNAS and see what it can do... I will also check my RAID Card and see if it can work with JBOD... fingers crossed... The machine has a couple internal SATA ports (think there are 2, could be 4) so i was thinking of using those for boot disks and SSDs later... As a

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Sašo Kiselkov
On 02/26/2013 09:33 AM, Tiernan OToole wrote: As a follow up question: Data Deduplication: The machine, to start, will have about 5Gb RAM. I read somewhere that 20TB storage would require about 8GB RAM, depending on block size... The typical wisdom is that 1TB of dedup'ed data = 1GB of RAM.

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Tim Cook
On Mon, Feb 25, 2013 at 10:33 PM, Tiernan OToole lsmart...@gmail.comwrote: Thanks all! I will check out FreeNAS and see what it can do... I will also check my RAID Card and see if it can work with JBOD... fingers crossed... The machine has a couple internal SATA ports (think there are 2, could

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Tiernan OToole
Thanks again lads. I will take all that info into advice, and will join that new group also! Thanks again! --Tiernan On Tue, Feb 26, 2013 at 8:44 AM, Tim Cook t...@cook.ms wrote: On Mon, Feb 25, 2013 at 10:33 PM, Tiernan OToole lsmart...@gmail.comwrote: Thanks all! I will check out

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Robert Milkowski
Solaris 11.1 (free for non-prod use). From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-boun...@opensolaris.org] On Behalf Of Tiernan OToole Sent: 25 February 2013 14:58 To: zfs-discuss@opensolaris.org Subject: [zfs-discuss] ZFS Distro Advice Good morning all. My home

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Gary Driggs
On Feb 26, 2013, at 12:44 AM, Sašo Kiselkov wrote: I'd also recommend that you go and subscribe to z...@lists.illumos.org, since this list is going to get shut down by Oracle next month. Whose description still reads, everything ZFS running on illumos-based distributions. -Gary

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Sašo Kiselkov
On 02/26/2013 03:51 PM, Gary Driggs wrote: On Feb 26, 2013, at 12:44 AM, Sašo Kiselkov wrote: I'd also recommend that you go and subscribe to z...@lists.illumos.org, since this list is going to get shut down by Oracle next month. Whose description still reads, everything ZFS running on

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Eugen Leitl
On Tue, Feb 26, 2013 at 06:51:08AM -0800, Gary Driggs wrote: On Feb 26, 2013, at 12:44 AM, Sašo Kiselkov wrote: I'd also recommend that you go and subscribe to z...@lists.illumos.org, since I can't seem to find this list. Do you have an URL for that? Mailman, hopefully? this list is going

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Sašo Kiselkov
On 02/26/2013 05:57 PM, Eugen Leitl wrote: On Tue, Feb 26, 2013 at 06:51:08AM -0800, Gary Driggs wrote: On Feb 26, 2013, at 12:44 AM, Sašo Kiselkov wrote: I'd also recommend that you go and subscribe to z...@lists.illumos.org, since I can't seem to find this list. Do you have an URL for

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Eugen Leitl
On Tue, Feb 26, 2013 at 06:01:39PM +0100, Sašo Kiselkov wrote: On 02/26/2013 05:57 PM, Eugen Leitl wrote: On Tue, Feb 26, 2013 at 06:51:08AM -0800, Gary Driggs wrote: On Feb 26, 2013, at 12:44 AM, Sašo Kiselkov wrote: I'd also recommend that you go and subscribe to z...@lists.illumos.org,

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Richard Elling
On Feb 26, 2013, at 12:33 AM, Tiernan OToole lsmart...@gmail.com wrote: Thanks all! I will check out FreeNAS and see what it can do... I will also check my RAID Card and see if it can work with JBOD... fingers crossed... The machine has a couple internal SATA ports (think there are 2, could

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Ian Collins
Robert Milkowski wrote: Solaris 11.1 (free for non-prod use). But a ticking bomb if you use a cache device. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Robert Milkowski
Robert Milkowski wrote: Solaris 11.1 (free for non-prod use). But a ticking bomb if you use a cache device. It's been fixed in SRU (although this is only for customers with a support contract - still, will be in 11.2 as well). Then, I'm sure there are other bugs which are fixed in

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Ian Collins
Robert Milkowski wrote: Robert Milkowski wrote: Solaris 11.1 (free for non-prod use). But a ticking bomb if you use a cache device. It's been fixed in SRU (although this is only for customers with a support contract - still, will be in 11.2 as well). Then, I'm sure there are other bugs

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Bob Friesenhahn
On Tue, 26 Feb 2013, Gary Driggs wrote: On Feb 26, 2013, at 12:44 AM, Sašo Kiselkov wrote: I'd also recommend that you go and subscribe to z...@lists.illumos.org,  since this list is going to get shut down by Oracle next month. Whose description still reads, everything ZFS

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Bob Friesenhahn
On Tue, 26 Feb 2013, Richard Elling wrote: Consider using different policies for different data. For traditional file systems, you had relatively few policy options: readonly, nosuid, quota, etc. With ZFS, dedup and compression are also policy options. In your case, dedup for your media is not

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Ian Collins
Bob Friesenhahn wrote: On Tue, 26 Feb 2013, Richard Elling wrote: Consider using different policies for different data. For traditional file systems, you had relatively few policy options: readonly, nosuid, quota, etc. With ZFS, dedup and compression are also policy options. In your case,

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Bob Friesenhahn
On Wed, 27 Feb 2013, Ian Collins wrote: I am finding that rsync with the right options (to directly block-overwrite) plus zfs snapshots is providing me with pretty amazing deduplication for backups without even enabling deduplication in zfs. Now backup storage goes a very long way. We do the

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Ian Collins
Bob Friesenhahn wrote: On Wed, 27 Feb 2013, Ian Collins wrote: I am finding that rsync with the right options (to directly block-overwrite) plus zfs snapshots is providing me with pretty amazing deduplication for backups without even enabling deduplication in zfs. Now backup storage goes a

Re: [zfs-discuss] ZFS Distro Advice

2013-02-26 Thread Jim Klimov
On 2013-02-27 05:36, Ian Collins wrote: Bob Friesenhahn wrote: On Wed, 27 Feb 2013, Ian Collins wrote: I am finding that rsync with the right options (to directly block-overwrite) plus zfs snapshots is providing me with pretty amazing deduplication for backups without even enabling

Re: [zfs-discuss] ZFS Distro Advice

2013-02-25 Thread Volker A. Brandt
Hi Tiernan! But, now i am confused as to what OS to use... OpenIndiana? Nexenta? FreeNAS/FreeBSD? I need something that will allow me to share files over SMB (3 if possible), NFS, AFP (for Time Machine) and iSCSI. Ideally, i would like something i can manage easily and something that works

Re: [zfs-discuss] ZFS Distro Advice

2013-02-25 Thread Suleyman Nazif Kutlu
I just started the same migration.. I tried to test OpenIndiana w/Napp-IT, Illumos w/Napp-IT, Debian k/FreeBSD and Nexenta Community Edition. I tested iSCSI, SMB and NFS. I did not find time to test FreeNAS. All tests performed as a VM guest on ESXi 5.0. SMB is based on kernel on OpenIndiana,

Re: [zfs-discuss] ZFS Distro Advice

2013-02-25 Thread Tim Cook
On Mon, Feb 25, 2013 at 4:57 AM, Tiernan OToole lsmart...@gmail.com wrote: Good morning all. My home NAS died over the weekend, and it leaves me with a lot of spare drives (5 2Tb and 3 1Tb disks). I have a Dell Poweredge 2900 Server sitting in the house, which has not been doing much over

Re: [zfs-discuss] ZFS Distro Advice

2013-02-25 Thread Volker A. Brandt
Tim Cook writes: I need something that will allow me to share files over SMB (3 if possible), NFS, AFP (for Time Machine) and iSCSI. Ideally, i would like something i can manage easily and something that works with the Dell... All of them should provide the basic functionality you're

Re: [zfs-discuss] ZFS Distro Advice

2013-02-25 Thread Tim Cook
On Mon, Feb 25, 2013 at 7:57 AM, Volker A. Brandt v...@bb-c.de wrote: Tim Cook writes: I need something that will allow me to share files over SMB (3 if possible), NFS, AFP (for Time Machine) and iSCSI. Ideally, i would like something i can manage easily and something that works with