Rsync is probably i bad idea yes. I could btrfs send -p the changed
"new" master subvolume and then delete the old master subvolume and then
reference the new master subvolume when transferring it later on i guess?
I'll explain the problem I'm trying to solve abit better;
Say i have a program that will run in multiple instances. The program
requires a dataset of large files to run (say 20GB). The dataset will be
updated over time, i.e parts of them changes. These changes should only
apply to new instances for the program. The program will also generate
new data (both new files and also changing data in the the shared
dataset) that is unique to the instance of the child subvolume. Finally
I need to transfer the program together with its generated data to
another remote machine to continue it's processing there. What i want to
achieve is avoid having to transfer the entire dataset when only small
parts of it is changed by the program. I also want to avoid having to
duplicate copies of the data on the remote machine.
On 2019-02-19 01:16, Chris Murphy wrote:
On Mon, Feb 18, 2019 at 4:58 PM André Malm <ad...@sheepa.org> wrote:
I assume i would have to use rsync (with --inplace possibly) to keep the
master volume in sync between machines?
Why?
You previously said you didn't want to do that:
" if I change / remove, say 10 GB worth of data from the
master subvolume unrelated to the child subvolume I don't want those
gigabytes sent down the wire with btrfs send as they are unrelated. "
So instead of sending those changes with send you're going to send the
changes with rsync which is even more inefficient?
Say for example I have a (large) file on master, on machine A, I cp
reflink it to a child subvolume. I then send -p child subvolume to
remote machine B (which already have the master volume).
1. The master snapshots must be identical snapshots on source and
destination; it must have been replicated using send/receive or
receive will complain.
2. I don't know what "send -p child subvolume to remove" means because
-p option means two snapshots must be included and I don't know what
two snapshots you're planning on using, so I can't answer the
question.
Then i change
parts of the file on the master of machine A. I then rsync (?) the
master volume so its the same across the machines. Can I then later send
-p the child volume, either back to the original machine (A) or to a 3rd
machine (C) given that the master volumes are synced?
rsync'd subvolumes across volumes aren't consistent identical by btrfs
receive. I expect they can't be used for incremental send/receive.
What you're trying to achieve, big picture, isn't clear. You're
describing it in terms of send/receive and confusing it with rsync. I
don't understand what rsync has to do with this.
You realize if you merely move a large file within a subvolume, and
then you using rsync to a remote system, it will delete the original
file and then sync over the nework that same data to the new location?
Rsync has no idea how to just move files around. Btrfs send-receive
does.
About the efficiency I'm not planning to remove large amounts of data
that is used by child subvolumes (although some will be updated). But
given the unpredictability of what files will be used by child
subvolumes i might remove large unused amounts of data.
OK?
