On Fri, Jul 6, 2012 at 12:34 AM, Goffredo Baroncelli <kreij...@libero.it> wrote:
> On 07/05/2012 06:51 PM, Alexander Block wrote:
>> Hello all,
>>
>> in IRC we had a discussion on how we could solve sending live
>> subvolumes and how to send subvolumes without the need to
>> administrate/keep old snapshots for incremental sends. One of the
>> ideas was to introduce "sendshots", which are basically snapshots
>> where no refs are counted for file data. This means, that when file
>> data is changed in the sendshot origin, we do not consume extra space
>> for two copies of the data. We would only have the metadata
>> duplicated.
>>
>> For the initial btrfs send we could do this:
>> 1. Create a hidden read-only snapshot of the subvolume to send. Hidden
>> means that it's not referenced by any subvolume. It is however still a
>> normal snapshot (not a sendshot!). Hidden snapshots are not possible
>> atm so we would have to implement that. This step allows us to send
>> read-write subvolumes, because we have a freezed version of it.
>
> Why we should want/need an hidden snapshot ? We could put this kind of
> hidden snapshot under a directory dot-prefixed (like /.hidden-subvolumes)
That would have the problem that the user may modify the subvolume
in-between (by removing the ro flag). Or he could simple cd into it
and we would later fail to delete it.
>
>> 2. Send this new snapshot.
>> 3. When we're done with sending, create a "sendshot" from the snapshot
>> and delete the invisible snapshot. As an alternative, we could convert
>> the invisible snapshot to a sendshot...but not sure if that would be
>> easy to implement.
>>
>> When we later do an incremental send we can do this:
>> 1. Do the same as point 1. from above.
>> 2. Determine which of the previous sendshots is the correct one for
>> the incremental send. We could use some magic auto detection here or
>> the user has to specify it by himself.
>> 3. Use the hidden snapshot from 1. and the determined sendshot from 2.
>> to find the incremental changes and do the send.
>
> I can understand how a sendshot could be used to compute the metadata
> delta. But how compute the data delta ?
We still would have the file extent data found in the metadata. When
we see that logical addresses or generations have changed, we know the
data has changed. This may however be problematic in case a defrag or
balance was performed, for this we should probably introduce a data
only transid or something like that which is preserved on such
operations.
>
>> 4. Do the same as point 3. from above.
>>
>> Every incremental send will add a new sendshot for later use. To avoid
>> having millions of such sendshots after some time, btrfs-progs would
>> need to delete old ones. That's something the user needs 100% control
>> of, as only he knows which ones can be deleted. He could either delete
>> them by hand or let btrfs send do that automatically with a parameter
>> that for example says how much sendshots to keep.
>>
>> The above steps would already make the use of btrfs send/receive a bit
>> easier. The next step would be to implement a network protocol that
>> allows on-the-fly sending/receiving without piping to a file
>> in-between. The protocol would allow the sending and receiving side to
>> agree on the sendshot to use for the incremental send. It would also
>> allow the sending side to do all the sendshot cleanups on its own,
>> because it would know which state is present on the receiving side.
>>
>> What do you guys think? Problem is, I probably won't be able to
>> implement this due to missing time for the rest of this year...going
>> on a world trip and I don't know when I'm back :)
>> So, if anyone wants too take this idea and implement it, feel free to do so
>> :)
>>
>> Alex.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> .
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
