btrfs send does not work from readonly device
Let's say we have a snapshot called snapshot on our device /dev/sdb. Now we boot a qemu machine and attatch the disk read only (since we do not want to put our data at risk in the vm, and read only should be enough to get our snapshot data). I used qemu-system-x86_64 -drive -file=/dev/sdb,if=scsi,readonly -hda path_to_guest_operating_system When we are in the qemu instance (suppose our device /dev/sdb is mounted on /mnt/import), we want to import our data now using btrfs send/receive: btrfs send /mnt/import/snapshot | btrfs receive / we get the following problems in syslog due to the fact that our device is read only: delayed_refs has NO entry BTRFS warning (device sdb): btrfs_update_root:151: Aborting unsused transaction(No space left) BTRFS warning (device sdb): Skipping commit of aborted transaction. BTRFS warning (device sdb): cleanup_transaction:1547: Aborting unused transaction (No space left) Apart from the snapshot folder being created in / nothing happens, the process simply hangs forever, with no actual data getting imported. Is that supposed to be that way? Why is readonly not enough to import data using btrfs send? -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Couple of problems regarding btrfs qgroup show reliability
Here is an simplified excerpt of my backup bash script:
CURRENT_TIME=$(date +%Y-%m-%d_%H:%M-%S)
# LAST_TIME variable contains the timestamp of the last backup in the same
format as $CURRENT_TIME
btrfs subvolume snapshot -r /mnt/root/@home /mnt/root/@home-
backup-$CURRENT_TIME
sync
# Define space check variables
btrfs quota enable /mnt/root
SUBVOLUME_ID=$(btrfs subvolume list /mnt/root | grep $CURRENT_TIME | awk
'{print $2}')
ABSOLUTE_SIZE=$(btrfs qgroup show /mnt/root | grep 0/$SUBVOLUME_ID | awk
'{print $2}')
RELATIVE_SIZE=$(btrfs qgroup show /mnt/root | grep 0/$SUBVOLUME_ID | awk
'{print $3}')
FREE_SPACE=$(df -B1 /mnt/backup | tail -1 | awk '{print $4}')
# Now I want to check if there is enough space on /mnt/backup, for sending the
incremental part to /mnt/backup (Let us assume us, that there have not been
made snapshots more recent than @home-backup-$LAST_TIME), so I did the
following in my backup script:
if (( $FREE_SPACE $RELATIVE_SIZE )); then
btrfs send -p /mnt/root/@home-backup-$LAST_TIME /mnt/root/@home-
backup-$CURRENT_TIME | btrfs receive /mnt/backup
fi
# For the initial bootstrapping I choose
if (( $FREE_SPACE $ABSOLUTE_SIZE )); then
btrfs send /mnt/root/@home-backup-$CURRENT_TIME | btrfs receive /mnt/backup
fi
Now I have a couple of questions:
1.) does it matter when I enable btrfs quota? I mean even if it is enabled for
the first time in the backup script? Does this have any influence on the values
determined for $ABSOLUTE_SIZE and $RELATIVE_SIZE?
2.) does btrfs implement some way to show free space on its own or do I have
to rely on df?
3.) Is the logic right for the incremental backup space check? I mean the
unshared space should be more or less what is transmittted by btrfs send,
right, since we already have the last snapshot on the backup drive? If this
isn't the right approach, how do I get the size difference of two special
snapshots say @home-backup-$CURRENT_TIME and @home-backup-$LAST_TIME?
4.) Out of curiosity I checked the ABSOLUTE_SIZE values of the sent snapshot
on the backup device too, in theory they should be equal right? But they are
not for some reason they are not equal at all, neither are the RELATIVE_SIZE
values.
Checking the ABSOLUTE_SIZE with du, seems to inidicate that the values on the
backup device seems to be right (2,6 GB), but on the internal drive the value
of $ABSOLUTE_SIZE is 2.0 GB, how can that be?
Of course the RELATIVE_SIZE can very a bit, depending on what snapshots are
residing on the same drive, but let as assume no confounding factors, then
they should be roughly in the same magnitude. And the ABSOLUTE_SIZE variables
should be definitely equal on both the backup drive and the internal harddrive
for the same snapshot.
Where am I wrong?
5.) I understand that btrfs snapshot delete breaks the RELATIVE_SIZE, at least
this is noted in the wiki. Is this still true, and will it be resolved soon?
The wiki also notes After deleting a subvolume, you must manually delete the
associated qgroup. How would I do that?
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
btrfs send handle missing space on target device
Hi, I wonder how I should handle missing space on the backup device when doing incremental backup with btrfs send? I mean for the initial bootstrap it is easy, I simply compare the free space of the target device with the size of home (we are talking about incremental backups of home), but how should I handle this with sudo btrfs send -p @home-backup @home-backup | sudo btrfs receive /mnt/backup/ What will happen by default if the space on the target is not enough? Will the command start writing to /mnt/backup/ at all and stop when the disk is full, or will it return an error code? If it doesn't return an error code, how can I handle this before the transaction? Thanks in advance! -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Backup: Compare sent snapshots
Hi, I am doing backups regularly following the scheme of https://btrfs.wiki.kernel.org/index.php/Incremental_Backup It states we keep a local reference of the read only snapshot we sent to the backup drive, which I understand. But now I have a question: When I do a read only snapshot of home, send the difference to the backup drive, keep it until the next incremental step, send the difference to the backup drive, remove the old read only snapshot and so on... I wonder what happens if the read only snapshot I keep as a local reference got corrupted somehow. Then maybe too much difference would be sent which would not be dramatic, or too less, which would be. Is there a quick way I could compare the last sent snapshot to the local one, to make sure the local reference is still the same? Apart from that, imagine I somehow lost the local reference (e.g. delete it by mistake), would there still be a way to sync the difference to the last sent snapshot on the backup device? -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Incremental backup over writable snapshot
Does anyone have a technical info regarding the reliability of the incremental backup process using the said method? (Apart from all the recommendations not to do it that way) So the question I am interested in: Should it work or not? I did some testing myself and it seemed to work, however I cannot find out if it backs up unnecessary blocks and thus making the incremental step space inefficient. That information would help me very much! Thank you very much! On Wednesday 19 February 2014 14:45:57 GEO wrote: Hi, As suggested in another thread, I would like to know the reliability of the following backup scheme: Suppose I have a subvolume of my homedirectory called @home. Now I am interested in making incremental backups of data in home I am interested in, but not everything, so I create a normal snapshot of @home called @home-w and delete the files/folders I am not interested in backing up. After that I create a readonly snapshot of @home-w called @home-r, that I sent to my target volume with btrfs send. After that is done, I do regular backups, by always going over the writeable snapshot where I remove always the same directories I am not interested and send the difference to the target volume with btrfs send -p @home-r @home-r-1| btrfs receive /path/of/target/volume. I do not like the idea of making subvolumes of all directories I am not interested in backing up. So what I would like to know now is the following: Could there be drawbacks of doing this resp. could I further optimize my backup strategy, as I experienced it takes a while for deleting large files in the writeable snapshot (What does it write there?) Could my method somehow lead to inefficiency in terms of the disk space used at the target volume (I mean, could the deleting cause a change, so that more is actually transferred as change, than in reality is?)? One last question would be: Is there a quick way I could verify the local read only snapshot used last time is the same as the one synced to the target volume last time? Thank you for your support and the great work! -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Incremental backup over writable snapshot
@Kai, Thank you very much for your reply. Sorry, I just saw it now. I will take care of the mailing issue now, so that it does not happen again in the future. Sorry for the inconveniences! -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Incremental backup over writable snapshot
First of all, I am sorry that I screw up the whole structure of the discussion (I have not subscribed to the mailing list, and as Kai replied to the mailing list only, I could not reply to his answer.) Kai: Yeah, your point was neutral and I did never understand it otherwise. Thank you for your answer! I already had idea of creating a subvolume called DATA in my home directory, however I find that pretty annoying, as most applications will open home by default. In fact I would find it more elegant to generally backup without making changes to my file system structure in home. I know that there other possibilities to do what I want, but I am asking whether the initially described method would work reliably, given that the user makes not a fundamental mistake himself. I know it may sound stubborn but I am really interested if my method works just as reliable as the other suggested methods. As I am not having the level required to check the code myself, I am asking here, in the hope someone knowing the code could state if it should work or not. Thank you all for your help! -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Incremental backup over writable snapshot
Hi, As suggested in another thread, I would like to know the reliability of the following backup scheme: Suppose I have a subvolume of my homedirectory called @home. Now I am interested in making incremental backups of data in home I am interested in, but not everything, so I create a normal snapshot of @home called @home-w and delete the files/folders I am not interested in backing up. After that I create a readonly snapshot of @home-w called @home-r, that I sent to my target volume with btrfs send. After that is done, I do regular backups, by always going over the writeable snapshot where I remove always the same directories I am not interested and send the difference to the target volume with btrfs send -p @home-r @home-r-1| btrfs receive /path/of/target/volume. I do not like the idea of making subvolumes of all directories I am not interested in backing up. So what I would like to know now is the following: Could there be drawbacks of doing this resp. could I further optimize my backup strategy, as I experienced it takes a while for deleting large files in the writeable snapshot (What does it write there?) Could my method somehow lead to inefficiency in terms of the disk space used at the target volume (I mean, could the deleting cause a change, so that more is actually transferred as change, than in reality is?)? One last question would be: Is there a quick way I could verify the local read only snapshot used last time is the same as the one synced to the target volume last time? Thank you for your support and the great work! -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Incremental backup over writable snapshot
On Wednesday 19 February 2014 10:00:49 Chris Murphy wrote: On Feb 19, 2014, at 6:45 AM, GEO 1g2e...@gmail.com wrote: I do not like the idea of making subvolumes of all directories I am not interested in backing up. Why? It addresses your use case. Chris Murphy I would prefer the idea of not snapshotting every directory I do not want to include, as there are almost more that I am not interested in. My question would simply be: Does the method going over the writeable snapshot and deleting things always lead to the same incremental end result as marking directories as snapshots that I am not interested in (apart from the additional empty directories created in case of the latter)? Furthermore hidden directories in home change very often, meaning if I install additional software, additional hidden directories may be created. So my script would have to mark them as snapshots every time. If I have hidden files, I cannot mark files as snapshots, so it is clear that my method makes sense. Once I have marked these directories snapshots and I want to create snapshots of my whole home subvolume I would always additionally have to specify those. So it makes the whole situation less manageable. Apart from that I find marking every directory I am not interested in as snapshots highly inelegant. So my question would be, if my preferred method is as reliable as the suggested method. Hope that's on the mailing list now :-). Thanks -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
BTRFS send: exclude directories
Hi, I have been experimenting with btrfs send/receive for incremental backups, but could not figure out how to exclude certain directories from subvolumes/snapshots. For example, I want to backup my data in home, but I am not interested in backing up hidden directories like .cache too. Is it currently possible to achieve something like that? Thanks for the great work! -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
