Re: [zfs-discuss] (Incremental) ZFS SEND at sub-snapshot level

2011-11-04 Thread Matthew Ahrens
On Sat, Oct 29, 2011 at 10:57 AM, Jim Klimov jimkli...@cos.ru wrote:

 In short, is it
 possible to add restartability to ZFS SEND


In short, yes.

We are working on it here at Delphix, and plan to contribute our changes
upstream to Illumos.

You can read more about it in the slides I link to in this blog post:

http://blog.delphix.com/matt/2011/11/01/zfs-10-year-anniversary/

--matt
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] (Incremental) ZFS SEND at sub-snapshot level

2011-10-31 Thread Paul Kraus
On Sat, Oct 29, 2011 at 1:57 PM, Jim Klimov jimkli...@cos.ru wrote:

  I am catching up with some 500 posts that I skipped this
 summer, and came up with a new question. In short, is it
 possible to add restartability to ZFS SEND, for example
 by adding artificial snapshots (of configurable increment
 size) into already existing datasets [too large to be
 zfs-sent successfully as one chunk of stream data]?

 We addressed this by decreasing our snapshot interval from 1 day
to 1 hour. We rarely have a snapshot bigger than a few GB now. I keep
meaning to put together a snapshot script that takes a new snapshot
when the amount of changed data increases to a certain point (for
example, take a snapshot whenever the snapshot would contain 250 MB of
data). Not enough round toits with all the other broken stuff to fix
:-(

-- 
{1-2-3-4-5-6-7-}
Paul Kraus
- Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
- Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
- Technical Advisor, RPI Players
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] (Incremental) ZFS SEND at sub-snapshot level

2011-10-30 Thread Jim Klimov

2011-10-30 2:14, Edward Ned Harvey пишет:

From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
boun...@opensolaris.org] On Behalf Of Jim Klimov

summer, and came up with a new question. In short, is it
possible to add restartability to ZFS SEND, for example


Rather than building something new and special into the filesystem, would
something like a restartable/continuable mbuffer command do the trick?


Well, it is true that for the purposes of sending a replication
stream over a flaky network, some sort of restartable buffer
program might suffice.

If one or both machines were rebooted in the process, however,
this would get us into the situation that all incomplete-snapshot
data was sent in vain, and the receiver has to destroy that data,
which may even get it to crash during pool import. Afterwards
the send attempt has to be done again, and if the conditions
were such that any attempt is likely to fail - it likely will.
Not all of our machines live in ivory-tower datacenters ;)

Per Paul Kraus (who recently wrote about similar problems):
 Uhhh, not being able to destroy snapshots that are too big
 is a pretty big one for us

Inserting artificial snapshots into existing datasets (perhaps
including the inheritance tree of huge incomplete snapshots
such as we can see now) might also allow to destroy an unneeded
dataset with less strain on the system, piece by piece. Perhaps
even without causing a loop of kernel panics, wow! ;)

The way I see it, this feature would help solve at least two
problems (or work-around them). To me these problems are
substantial. Perhaps to others, like Paul, too.

Because of highly-probable failures during a single unit of
ZFS-SEND replication, I am bound to not use it at all.
I also have to plan destruction of datasets at my home rig
(which was tainted with dedup) and expect weeks of downtime
while the system is being reset to crawl through the blocks
being released after a large delete...

//Jim
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] (Incremental) ZFS SEND at sub-snapshot level

2011-10-30 Thread Jim Klimov

2011-10-29 21:57, Jim Klimov пишет:

... In short, is it
possible to add restartability to ZFS SEND, for example
by adding artificial snapshots (of configurable increment
size) into already existing datasets [too large to be
zfs-sent successfully as one chunk of stream data]?




On a side note: would this feature, like any other nice-to-have
feature in ZFS, require The Mythical Block Pointer Rewrite (TM)?

For no apparent reason yet, I'm already afraid so ;)

If this is the Holy Grail which everybody craves and nobody saw,
what is really the problem of making it happen? Some time ago
I skimmed through an overview of what would have to be done
for it. Not being a hardcore ZFS programmer I did not grasp
what is so fundamentally difficult about the quest. So I still
wonder if it is impossible, or if anyone is already working
on it quietly? ;)

//Jim
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss


Re: [zfs-discuss] (Incremental) ZFS SEND at sub-snapshot level

2011-10-29 Thread Edward Ned Harvey
 From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
 boun...@opensolaris.org] On Behalf Of Jim Klimov
 
 summer, and came up with a new question. In short, is it
 possible to add restartability to ZFS SEND, for example

Rather than building something new and special into the filesystem, would
something like a restartable/continuable mbuffer command do the trick?  It
seems to be a general issue, not filesystem specific - that you want to
tunnel some command or some data stream through a buffering (perhaps even
checksumming/error detecting/correcting) buffering system, to make it more
resilient crossing a WAN or whatever.

There is probably already a utility like that.  I quickly checked mbuffer to
see if it did, but it didn't seem to do that.  I didn't look very deeply, I
could be wrong.

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss