[zfs-discuss] Best way/issues with large ZFS send?
I'm preparing to replicate about 200TB of data between two data centers using zfs send. We have ten 10TB zpools that are further broken down into zvols of various sizes in each data center. One DC is primary and the other will be the replication target and there is plenty of bandwidth between them (10 gig dark fiber). Are there any gotchas that I should be aware of? Also, at what level should I be taking the snapshot to do the zfs send? At the primary pool level or at the zvol level? Since the targets are to be exact replicas, I presume at the primary pool level (e.g. tank) rather than for every zvol (e.g. tank/prod/vol1)? This is all using Solaris 11 Express, snv_151a. Thanks, Eff -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Best way/issues with large ZFS send?
On Feb 16, 2011, at 6:05 AM, Eff Norwood wrote: I'm preparing to replicate about 200TB of data between two data centers using zfs send. We have ten 10TB zpools that are further broken down into zvols of various sizes in each data center. One DC is primary and the other will be the replication target and there is plenty of bandwidth between them (10 gig dark fiber). Are there any gotchas that I should be aware of? Also, at what level should I be taking the snapshot to do the zfs send? At the primary pool level or at the zvol level? Since the targets are to be exact replicas, I presume at the primary pool level (e.g. tank) rather than for every zvol (e.g. tank/prod/vol1)? There is no such thing as a pool snapshot. There are only dataset snapshots. The trick to a successful snapshot+send strategy at this size is to start snapping early and often. You don't want to send 200TB, you want to send 2TB, 100 times :-) The performance is tends to be bursty, so the fixed record size of the zvols can work to your advantage for capacity planning. Also, a buffer of some sort can help smooth out the utilization, see the threads on ZFS and mbuffer. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Best way/issues with large ZFS send?
From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Eff Norwood Are there any gotchas that I should be aware of? Also, at what level should I be taking the snapshot to do the zfs send? At the primary pool level or at the zvol level? Since the targets are to be exact replicas, I presume at the primary pool level (e.g. tank) rather than for every zvol (e.g. tank/prod/vol1)? I don't think you have any choice. There is such a thing as zfs send and there is no such thing as zpool send. You can use -R for replication recursive, which is kind of like doing the zpool directly... But not exactly. The only gotcha is to ensure your applications (whatever they are) are resilient to power failure. Because the zfs snapshot will literally produce a block-level snapshot of the way the disks are right now, at this instant. If you ever need to restore, then it will be as-if you sustained a power loss and came back online. The most notable such situation would probably be databases, if you have any. Ensure to use a database backup tool to export the database to a backup file, and then send the filesystem. Or else momentarily stop the database services while you take the snapshot. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Best way/issues with large ZFS send?
On 16 February, 2011 - Richard Elling sent me these 1,3K bytes:
On Feb 16, 2011, at 6:05 AM, Eff Norwood wrote:
I'm preparing to replicate about 200TB of data between two data centers
using zfs send. We have ten 10TB zpools that are further broken down into
zvols of various sizes in each data center. One DC is primary and the other
will be the replication target and there is plenty of bandwidth between
them (10 gig dark fiber).
Are there any gotchas that I should be aware of? Also, at what level should
I be taking the snapshot to do the zfs send? At the primary pool level or
at the zvol level? Since the targets are to be exact replicas, I presume at
the primary pool level (e.g. tank) rather than for every zvol (e.g.
tank/prod/vol1)?
There is no such thing as a pool snapshot. There are only dataset snapshots.
.. but you can make a single recursive snapshot call that affects all
datasets.
The trick to a successful snapshot+send strategy at this size is to start
snapping
early and often. You don't want to send 200TB, you want to send 2TB, 100
times :-)
The performance is tends to be bursty, so the fixed record size of the zvols
can
work to your advantage for capacity planning. Also, a buffer of some sort can
help
smooth out the utilization, see the threads on ZFS and mbuffer.
-- richard
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
/Tomas
--
Tomas Ögren, st...@acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Umeå
`- Sysadmin at {cs,acc}.umu.se
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
