Re: [zfs-discuss] (Incremental) ZFS SEND at sub-snapshot level
On Sat, Oct 29, 2011 at 10:57 AM, Jim Klimov wrote: > In short, is it > possible to add "restartability" to ZFS SEND In short, yes. We are working on it here at Delphix, and plan to contribute our changes upstream to Illumos. You can read more about it in the slides I link to in this blog post: http://blog.delphix.com/matt/2011/11/01/zfs-10-year-anniversary/ --matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] (Incremental) ZFS SEND at sub-snapshot level
On Sat, Oct 29, 2011 at 1:57 PM, Jim Klimov wrote:
> I am catching up with some 500 posts that I skipped this
> summer, and came up with a new question. In short, is it
> possible to add "restartability" to ZFS SEND, for example
> by adding artificial snapshots (of configurable increment
> size) into already existing datasets [too large to be
> zfs-sent successfully as one chunk of stream data]?
We addressed this by decreasing our snapshot interval from 1 day
to 1 hour. We rarely have a snapshot bigger than a few GB now. I keep
meaning to put together a snapshot script that takes a new snapshot
when the amount of changed data increases to a certain point (for
example, take a snapshot whenever the snapshot would contain 250 MB of
data). Not enough round toits with all the other broken stuff to fix
:-(
--
{1-2-3-4-5-6-7-}
Paul Kraus
-> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
-> Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
-> Technical Advisor, RPI Players
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] (Incremental) ZFS SEND at sub-snapshot level
2011-10-29 21:57, Jim Klimov пишет: ... In short, is it possible to add "restartability" to ZFS SEND, for example by adding artificial snapshots (of configurable increment size) into already existing datasets [too large to be zfs-sent successfully as one chunk of stream data]? On a side note: would this feature, like any other nice-to-have feature in ZFS, require The Mythical Block Pointer Rewrite (TM)? For no apparent reason yet, I'm already afraid so ;) If this is the Holy Grail which everybody craves and nobody saw, what is really the problem of making it happen? Some time ago I skimmed through an overview of "what would have to be done for it". Not being a hardcore ZFS programmer I did not grasp what is so fundamentally difficult about the quest. So I still wonder if it is impossible, or if anyone is already working on it quietly? ;) //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] (Incremental) ZFS SEND at sub-snapshot level
2011-10-30 2:14, Edward Ned Harvey пишет: From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Jim Klimov summer, and came up with a new question. In short, is it possible to add "restartability" to ZFS SEND, for example Rather than building something new and special into the filesystem, would something like a restartable/continuable mbuffer command do the trick? Well, it is true that for the purposes of sending a replication stream over a flaky network, some sort of restartable buffer program might suffice. If one or both machines were rebooted in the process, however, this would get us into the situation that all incomplete-snapshot data was sent in vain, and the receiver has to destroy that data, which may even get it to crash during pool import. Afterwards the send attempt has to be done again, and if the conditions were such that any attempt is likely to fail - it likely will. Not all of our machines live in ivory-tower datacenters ;) Per Paul Kraus (who recently wrote about similar problems): > Uhhh, not being able to destroy snapshots that are "too big" > is a pretty big one for us Inserting artificial snapshots into existing datasets (perhaps including the inheritance tree of "huge incomplete snapshots" such as we can see now) might also allow to destroy an unneeded dataset with less strain on the system, piece by piece. Perhaps even without causing a loop of kernel panics, wow! ;) The way I see it, this feature would help solve at least two problems (or work-around them). To me these problems are substantial. Perhaps to others, like Paul, too. Because of highly-probable failures during a single unit of ZFS-SEND replication, I am bound to not use it at all. I also have to plan destruction of datasets at my home rig (which was tainted with dedup) and expect weeks of downtime while the system is being reset to crawl through the blocks being released after a large delete... //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] (Incremental) ZFS SEND at sub-snapshot level
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Jim Klimov > > summer, and came up with a new question. In short, is it > possible to add "restartability" to ZFS SEND, for example Rather than building something new and special into the filesystem, would something like a restartable/continuable mbuffer command do the trick? It seems to be a general issue, not filesystem specific - that you want to tunnel some command or some data stream through a buffering (perhaps even checksumming/error detecting/correcting) buffering system, to make it more resilient crossing a WAN or whatever. There is probably already a utility like that. I quickly checked mbuffer to see if it did, but it didn't seem to do that. I didn't look very deeply, I could be wrong. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
