Re: [gentoo-user] BACKUPS
On 10/09/2014 07:32, Joseph wrote: Thank you again. On a different subject. Do you have a good pointer on how to backup a system. Did it even occur to you at all to type that exact question into a search engine and see what comes back? Come on dude, we aren't here to do ALL your thinking for you. -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] BACKUPS
Alan McKinnon wrote: On 10/09/2014 07:32, Joseph wrote: Thank you again. On a different subject. Do you have a good pointer on how to backup a system. Did it even occur to you at all to type that exact question into a search engine and see what comes back? Come on dude, we aren't here to do ALL your thinking for you. Even easier, eix backup ;-) Dale :-) :-)
Re: [gentoo-user] BACKUPS
On Wednesday, September 10, 2014 02:47:26 AM Dale wrote: Alan McKinnon wrote: On 10/09/2014 07:32, Joseph wrote: Thank you again. On a different subject. Do you have a good pointer on how to backup a system. Did it even occur to you at all to type that exact question into a search engine and see what comes back? Come on dude, we aren't here to do ALL your thinking for you. Even easier, eix backup That only returns 18 packages and you're missing most of the ones in app- backup/. Try: eix app-backup/ That gives you 44 to choose from. Alternatively, copy every file to a usb-disk and manually confirm they are all identical? -- Joost
Re: [gentoo-user] BACKUPS
J. Roeleveld wrote: On Wednesday, September 10, 2014 02:47:26 AM Dale wrote: Alan McKinnon wrote: On 10/09/2014 07:32, Joseph wrote: Thank you again. On a different subject. Do you have a good pointer on how to backup a system. Did it even occur to you at all to type that exact question into a search engine and see what comes back? Come on dude, we aren't here to do ALL your thinking for you. Even easier, eix backup That only returns 18 packages and you're missing most of the ones in app-backup/. Try: eix app-backup/ That gives you 44 to choose from. Alternatively, copy every file to a usb-disk and manually confirm they are all identical? -- Joost Well, I first used eix backup. My thinking was this, if one searches first for something with backup in it, they should notice a lot of apps in the app-backup category even tho they might not know it existed before that. At that point, pot of gold with this: eix app-backup/* Thing is, he may not have eix installed. :/ Either way works tho. :-D Me, I just use rsync and call it a day. Dale :-) :-)
Re: [gentoo-user] BACKUPS
On Wed, 10 Sep 2014 09:14:43 +0200, Alan McKinnon wrote: Thank you again. On a different subject. Do you have a good pointer on how to backup a system. Did it even occur to you at all to type that exact question into a search engine and see what comes back? And if you then need to ask a question on a different subject, please start a new thread. It's still thread-jacking if the thread you hijack is one you started. -- Neil Bothwick I am Flatulus of Borg. You will be asphixiated. signature.asc Description: PGP signature
Re: [gentoo-user] BACKUPS
On 10/09/2014 10:03, Dale wrote: J. Roeleveld wrote: On Wednesday, September 10, 2014 02:47:26 AM Dale wrote: Alan McKinnon wrote: On 10/09/2014 07:32, Joseph wrote: Thank you again. On a different subject. Do you have a good pointer on how to backup a system. Did it even occur to you at all to type that exact question into a search engine and see what comes back? Come on dude, we aren't here to do ALL your thinking for you. Even easier, eix backup That only returns 18 packages and you're missing most of the ones in app-backup/. Try: eix app-backup/ That gives you 44 to choose from. Alternatively, copy every file to a usb-disk and manually confirm they are all identical? -- Joost Well, I first used eix backup. My thinking was this, if one searches first for something with backup in it, they should notice a lot of apps in the app-backup category even tho they might not know it existed before that. At that point, pot of gold with this: eix app-backup/* Thing is, he may not have eix installed. :/ that's why we have emerge -s :-) Either way works tho. :-D Me, I just use rsync and call it a day. Dale :-) :-) -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] BACKUPS
On 10/09/2014 06:32, Joseph wrote: On 09/10/14 06:10, Kerin Millar wrote: snip Thank you again. On a different subject. Do you have a good pointer on how to backup a system. I just had a HD crash so I selected a replacement SSD and I'm re installing the software. I had backup of /etc/ and /home but I've missed all other settings eg: /boot/ kernel config and other files and are not in /etc directory like: hylafax setting etc. Is there a way to keep backup of all those configuration files that are manually edited? As suggested by Neil, please begin a new thread. --Kerin
Re: [gentoo-user] BACKUPS
On 09/10/14 06:10, Kerin Millar wrote: [snip] I just extracted the files with tar... I read your forum post and can see that you're (dangerously) extracting directly into the root directory and that this is among the contents of the archive: ./usr/lib/ ./usr/lib/libbrcomplpr2.so I posit that tar clobbers the /usr/lib symlink, converting it into a directory because that is what is stored in the archive. Ergo, use the --keep-directory-symlink parameter. Excuse the fact that I am replying to myself, but I must also stress that the library does not belong in lib64. On a 64-bit system, you should adapt your process so that the library ends up residing in lib32, not lib64 (by way of the lib symlink). The software will not be able to function correctly otherwise. --Kerin Thank you again. On a different subject. Do you have a good pointer on how to backup a system. I just had a HD crash so I selected a replacement SSD and I'm re installing the software. I had backup of /etc/ and /home but I've missed all other settings eg: /boot/ kernel config and other files and are not in /etc directory like: hylafax setting etc. Is there a way to keep backup of all those configuration files that are manually edited? -- Joseph
Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On Tuesday 27 May 2014 23:35:26 Alan McKinnon wrote: On 27/05/2014 17:12, J. Roeleveld wrote: I have a yearly (full), monthly, weekly and daily. Each incremental is against the most recent one of itself or longer period. That means having to keep multiple snapshots active, which I prefer to avoid. But, it is a good idea for backing up desktops and laptops. I'm curious why you have yearly snapshots. I've yet to find any sane production system where a yearly backup had any worth at all. Even monthly is pushing it... Or do you do it to have a decent start point for incrementals? It's to have a decent start point for incrementals. Below are the 2 biggest shares on the NAS: /dev/xvda17 7.1T 5.9T 1.2T 84% /data/unsorted /dev/xvda16 3.0T 2.4T 517G 83% /data/software It is impossible to do a full backup on a daily or even weekly basis. Previously, I had 1 full backup and then a daily incremental. This appears like a good idea, untill you need to restore the filesystem from backups when the crash occured 2 years later. That is 1 full backup and over 700 incrementals Currently, I do the following: Every year, a full backup Then, every month, I have an incremental based on either the yearly or previous monthly. Ditto for the weekly (but then based on monthly or weekly) And again for the daily. -- Joost
Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On Tuesday 27 May 2014 11:32:22 Rich Freeman wrote: On Tue, May 27, 2014 at 11:21 AM, J. Roeleveld jo...@antarean.org wrote: Does anyone know how these will handle (and perform) with a possible 300+ snapshots per filesystem (or volume, as I think it's called)? I can't speak for zfs. I had upwards of 1000 snapshots on my system before I stopped creating them hourly and and just started having issues with it. Ok, it can handle my backup schedule then. :) I wouldn't really say it is ready for prime time, but it is workable. Of course, you'll still want backups - a million snapshots does you no good if some bug wipes out your filesystem. For one of my ENOSPC incidents I ended up just wiping the entire filesystem and restoring from backup, though if I kept at it I'd probably have been able to fix it. Agreed, this is why the backup system would be adjusted for BTRFS on the fileserver, when I get round to it. I would probably keep snapshots active for the past 2 weeks. Oh, one other tip if you use btrfs - be sure you have a rescue disk that supports it. Hint, the old Gentoo install CD I had lying around didn't. You'll probably want to keep a rescue CD with a recent kernel and btrfs-tools handy at all times. Always important. I just saw the other email which states that the latest sysresccd supports it. That is fine for me. -- Joost
Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On Wed, 28 May 2014 12:03 +0200, Joost Roeleveld wrote: Always important. I just saw the other email which states that the latest sysresccd supports it. That is fine for me. Sysresccd has supported btrfs for some time. I realise my mail could have been read otherwise, but the reason for keeping a recent CD for btrfs is that it is evolving and the recommendation is to always use the latest kernel and userspace available. -- Neil Bothwick If you can smile when things go wrong, you have someone in mind to blame. signature.asc Description: PGP signature
Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On Tuesday 27 May 2014 11:28:17 Rich Freeman wrote: On Tue, May 27, 2014 at 11:12 AM, J. Roeleveld jo...@antarean.org wrote: On Tuesday, May 27, 2014 10:31:26 AM Rich Freeman wrote: btrfs wouldn't have any issues with this at all. You'd have an advantage in that you wouldn't have to unmount the filesystem to cleanly create the snapshot (which you have to do with lvm). That, or a sync prior to creating the snapshot. :) If the filesystem is still mounted, I'm not sure that a sync is guaranteed to give you a clean remount. It only flushes the caches/etc. You need to remount read-only or unmount before doing the sync (and the sync probably isn't actually necessary as I'd think LVM would snapshot the contents of the cache as well). I do this for the OS-partitions of my VMs. In the VM, run sync, then on the host, take an LVM snapshot and then mount the snapshot on the host. I have not had any errors from this. I have a yearly (full), monthly, weekly and daily. Each incremental is against the most recent one of itself or longer period. That means having to keep multiple snapshots active, which I prefer to avoid. You only need to store snapshots for use with incremental backups. So, if all your backups are full, then you don't need to retain any snapshots (and you wouldn't use btrfs send anyway). If your yearly is full and your monthlies are incremental against the yearly then you need to keep your yearly snapshot for a year. If your yearly is full and your monthlies are incremental against the last month, then you only need to keep the yearly until the next monthly. If your monthlies are full then you only need to keep the current monthly assuming your dailies are incremental against those, but if they're incremental from the last daily then you never need to keep anything for more than a day. That makes for an interesting option. Not sure if I would implement it that way. But, it is a good idea for backing up desktops and laptops. It is really intended more for something like datacenter replication. Snapshot every 5 min, send the data to the backup datacenter, delete the snapshots upon confirmation of successful receipt. In such a scenario you wouldn't retain the sent files but just keep playing them against the replicate filesystem. They'd be fine for backups as well, as long as you can store the snapshots online until no longer needed for incrementals. app-backup/dar uses catalogues for the incrementals. I think I will stick to that for the foreseeable future. But, you can always just create a snapshot, write it to backup with your favorite tool (it is just a directory tree), and then remove it as soon as you're done with it. Creating a snapshot is atomic at the filesystem level, though again if you want application level consistency you need to deal with that until somebody comes up with a transactional way to store files on Linux that is more elegant that fsyncing on every write. That would require a method to keep database and filesystem perfectly in sync when they are not necessarily on the same machine. Well, right now we can't even guarantee consistency when everything is written by a single process on the same machine. The best we have is a clunky fsync operation which kills the write cache and destroys performance, and even that doesn't do anything if you have more than one file that must be consistent. Yep, and that's why those filesystems are actually umounted prior to creating the LVM snapshot. Umounting forces the filesystem to be put into a consistent state. The result is journals on top of journals as nobody can trust the next layer down to do its job correctly. Going across machines does complicate things further as there are more failure modes that take out one part of the overall system but not another. However, I'd like to think that an OS that natively supports transactions could at least standardize things so that every layer along the path isn't storing its own journal. In fact, many of the optimizations possible with zfs and btrfs are due to the fact that you eliminate all those layers. Either of those 2, probably btrfs as I prefer native support in the kernel, will be implemented when I get the opportunity to get the NAS on bare metal and remove the virtualization for that component. I need to find a different host for the other services first. That might take a while. -- Joost
Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On Wed, May 28, 2014 at 6:13 AM, Joost Roeleveld jo...@antarean.org wrote: app-backup/dar uses catalogues for the incrementals. I think I will stick to that for the foreseeable future. I used to use that and sarab (which is a wrapper). I moved on to duplicity. The problem with dar is that it uses quite a bit of RAM as the number of files being backed up grows I think. So, if you have 6TB full of multimedia it might not be a huge problem, but if you have 6TB full of portage trees good luck with that. The other problem with dar is that if a file changes it stores a complete copy of it. Duplicity uses librsync, so if a file changes it only stores the parts that actually changed. It also uses catalogs, and supports things like caching catalogs (so you don't need the last incremental mounted), and a bunch of storage backends (like S3). However, dar definitely is more useful than tar if you want the option for random access. Rich
Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On 28/05/2014 11:58, Joost Roeleveld wrote: On Tuesday 27 May 2014 23:35:26 Alan McKinnon wrote: On 27/05/2014 17:12, J. Roeleveld wrote: I have a yearly (full), monthly, weekly and daily. Each incremental is against the most recent one of itself or longer period. That means having to keep multiple snapshots active, which I prefer to avoid. But, it is a good idea for backing up desktops and laptops. I'm curious why you have yearly snapshots. I've yet to find any sane production system where a yearly backup had any worth at all. Even monthly is pushing it... Or do you do it to have a decent start point for incrementals? It's to have a decent start point for incrementals. Below are the 2 biggest shares on the NAS: /dev/xvda17 7.1T 5.9T 1.2T 84% /data/unsorted /dev/xvda16 3.0T 2.4T 517G 83% /data/software It is impossible to do a full backup on a daily or even weekly basis. Previously, I had 1 full backup and then a daily incremental. This appears like a good idea, untill you need to restore the filesystem from backups when the crash occured 2 years later. That is 1 full backup and over 700 incrementals Currently, I do the following: Every year, a full backup Then, every month, I have an incremental based on either the yearly or previous monthly. Ditto for the weekly (but then based on monthly or weekly) And again for the daily. OK, that makes sense. It reminds me of an issue my wife had with the data warehouse when she worked at the bank. In a nutshell, they needed backups but backups were impossible to achieve because physics says so. They needed to get data off the disk 4 times faster than data comes off a disk - SCSI limits being rather hard limits :-) That opinion didn't go down well when I offered it The solution was to do it much like your plan above. With the benefit that the infrequent full backups would be done on a fixed schedule in a change window with X hours downtime that was known well in advance. -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On Wednesday 28 May 2014 13:07:49 Alan McKinnon wrote: On 28/05/2014 11:58, Joost Roeleveld wrote: On Tuesday 27 May 2014 23:35:26 Alan McKinnon wrote: On 27/05/2014 17:12, J. Roeleveld wrote: I have a yearly (full), monthly, weekly and daily. Each incremental is against the most recent one of itself or longer period. That means having to keep multiple snapshots active, which I prefer to avoid. But, it is a good idea for backing up desktops and laptops. I'm curious why you have yearly snapshots. I've yet to find any sane production system where a yearly backup had any worth at all. Even monthly is pushing it... Or do you do it to have a decent start point for incrementals? It's to have a decent start point for incrementals. Below are the 2 biggest shares on the NAS: /dev/xvda17 7.1T 5.9T 1.2T 84% /data/unsorted /dev/xvda16 3.0T 2.4T 517G 83% /data/software It is impossible to do a full backup on a daily or even weekly basis. Previously, I had 1 full backup and then a daily incremental. This appears like a good idea, untill you need to restore the filesystem from backups when the crash occured 2 years later. That is 1 full backup and over 700 incrementals Currently, I do the following: Every year, a full backup Then, every month, I have an incremental based on either the yearly or previous monthly. Ditto for the weekly (but then based on monthly or weekly) And again for the daily. OK, that makes sense. It reminds me of an issue my wife had with the data warehouse when she worked at the bank. In a nutshell, they needed backups but backups were impossible to achieve because physics says so. They needed to get data off the disk 4 times faster than data comes off a disk - SCSI limits being rather hard limits :-) That opinion didn't go down well when I offered it Haha :) I know the feeling. I'd love to know the final solution they came up with. The solution was to do it much like your plan above. With the benefit that the infrequent full backups would be done on a fixed schedule in a change window with X hours downtime that was known well in advance. Using snapshots, the downtime is the same couple of minutes each night. The problem is that during the backup, the performance of the server is impacted. For a full backup, that means weeks... -- Joost
Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On 28/05/2014 13:42, Joost Roeleveld wrote: Currently, I do the following: Every year, a full backup Then, every month, I have an incremental based on either the yearly or previous monthly. Ditto for the weekly (but then based on monthly or weekly) And again for the daily. OK, that makes sense. It reminds me of an issue my wife had with the data warehouse when she worked at the bank. In a nutshell, they needed backups but backups were impossible to achieve because physics says so. They needed to get data off the disk 4 times faster than data comes off a disk - SCSI limits being rather hard limits :-) That opinion didn't go down well when I offered it Haha :) I know the feeling. I'd love to know the final solution they came up with. It was clever magic with LVM snapshots, but I'm not privy to the details - it was a bank after all and there's only so much Mrs Alan and the sysadmins could tell me. But it went something like this: Take a snapshot and copy that data to the SAN. This takes days or weeks and it's only ever done once. Thereafter, snapshots only. The backup system would take that last full + incrementals and create a new full for it's own use, this process runs independent from everything else and must take as long as it takes while the db carries on doing it's thing. If two backup jobs start to overlap in time, one of them gets discarded. Quite a clever scheme actually and it relies on storage being shared on the SAN to work, plus some in-house magic to get the backup system to recognise and use LVM snapshots natively. I believe performance impact was kept to acceptable levels by cleverly setting priorities - read/writes by the db take priority over backup reads which has to take a back seat and just wait it's turn. The solution was to do it much like your plan above. With the benefit that the infrequent full backups would be done on a fixed schedule in a change window with X hours downtime that was known well in advance. Using snapshots, the downtime is the same couple of minutes each night. The problem is that during the backup, the performance of the server is impacted. For a full backup, that means weeks... -- Alan McKinnon alan.mckin...@gmail.com
[gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On Tuesday, May 27, 2014 10:31:26 AM Rich Freeman wrote: On Tue, May 27, 2014 at 10:09 AM, J. Roeleveld jo...@antarean.org wrote: I am still happily using LVM with snapshots. Those are instantaneous as well and I can then backup the snapshot, which on my server takes between 2 hours (incremental) and 3 weeks (full) When a snapshot is backed up, it is removed. The process to create the snapshots runs daily, but I could also configure it to run more often. This means that when I start a daily backup, the incrementals are piling up as snapshots. With 15 different filesystems to backup, I didn't experience any issue with this. I wonder how btrfs would deal with a situation like this? btrfs wouldn't have any issues with this at all. You'd have an advantage in that you wouldn't have to unmount the filesystem to cleanly create the snapshot (which you have to do with lvm). That, or a sync prior to creating the snapshot. :) If you're concerned about application-level consistency you still need to get applications to flush their writes/checkpoint/etc (which don't have to be on disk, but they do have to be sent to the kernel). Application-level consistency, for some of the filesystems, means stopping the application, taking a backup of the database, creating a snapshot and then restarting the application. For all the applications I run, the entire nightly process takes 2 minutes in total. During this time, services become temporarily unavailable. This is acceptable. If you want to get really crazy you could make use of btrfs send as well - which is a filesystem-level function which tracks the actual changes between snapshots. Think of it like librsync with full file comparisons (a very expensive mode that few use in practice) but it doesn't need to actually read the files or have access to the destination files to find the differences. Doing this does require keeping around a snapshot until all backups incrementally created against it are done (if there are going to be any). I have a yearly (full), monthly, weekly and daily. Each incremental is against the most recent one of itself or longer period. That means having to keep multiple snapshots active, which I prefer to avoid. But, it is a good idea for backing up desktops and laptops. But, you can always just create a snapshot, write it to backup with your favorite tool (it is just a directory tree), and then remove it as soon as you're done with it. Creating a snapshot is atomic at the filesystem level, though again if you want application level consistency you need to deal with that until somebody comes up with a transactional way to store files on Linux that is more elegant that fsyncing on every write. That would require a method to keep database and filesystem perfectly in sync when they are not necessarily on the same machine.
Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On Tuesday, May 27, 2014 05:12:50 PM J. Roeleveld wrote: On Tuesday, May 27, 2014 10:31:26 AM Rich Freeman wrote: On Tue, May 27, 2014 at 10:09 AM, J. Roeleveld jo...@antarean.org wrote: snipped Forgot to add: For fileservers, I am starting to feel that ZFS or BTRFS snapshots are easier to work with as it makes restoring files simpler. Does anyone know how these will handle (and perform) with a possible 300+ snapshots per filesystem (or volume, as I think it's called)? -- Joost
Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On Tue, May 27, 2014 at 11:12 AM, J. Roeleveld jo...@antarean.org wrote: On Tuesday, May 27, 2014 10:31:26 AM Rich Freeman wrote: btrfs wouldn't have any issues with this at all. You'd have an advantage in that you wouldn't have to unmount the filesystem to cleanly create the snapshot (which you have to do with lvm). That, or a sync prior to creating the snapshot. :) If the filesystem is still mounted, I'm not sure that a sync is guaranteed to give you a clean remount. It only flushes the caches/etc. You need to remount read-only or unmount before doing the sync (and the sync probably isn't actually necessary as I'd think LVM would snapshot the contents of the cache as well). I have a yearly (full), monthly, weekly and daily. Each incremental is against the most recent one of itself or longer period. That means having to keep multiple snapshots active, which I prefer to avoid. You only need to store snapshots for use with incremental backups. So, if all your backups are full, then you don't need to retain any snapshots (and you wouldn't use btrfs send anyway). If your yearly is full and your monthlies are incremental against the yearly then you need to keep your yearly snapshot for a year. If your yearly is full and your monthlies are incremental against the last month, then you only need to keep the yearly until the next monthly. If your monthlies are full then you only need to keep the current monthly assuming your dailies are incremental against those, but if they're incremental from the last daily then you never need to keep anything for more than a day. But, it is a good idea for backing up desktops and laptops. It is really intended more for something like datacenter replication. Snapshot every 5 min, send the data to the backup datacenter, delete the snapshots upon confirmation of successful receipt. In such a scenario you wouldn't retain the sent files but just keep playing them against the replicate filesystem. They'd be fine for backups as well, as long as you can store the snapshots online until no longer needed for incrementals. But, you can always just create a snapshot, write it to backup with your favorite tool (it is just a directory tree), and then remove it as soon as you're done with it. Creating a snapshot is atomic at the filesystem level, though again if you want application level consistency you need to deal with that until somebody comes up with a transactional way to store files on Linux that is more elegant that fsyncing on every write. That would require a method to keep database and filesystem perfectly in sync when they are not necessarily on the same machine. Well, right now we can't even guarantee consistency when everything is written by a single process on the same machine. The best we have is a clunky fsync operation which kills the write cache and destroys performance, and even that doesn't do anything if you have more than one file that must be consistent. The result is journals on top of journals as nobody can trust the next layer down to do its job correctly. Going across machines does complicate things further as there are more failure modes that take out one part of the overall system but not another. However, I'd like to think that an OS that natively supports transactions could at least standardize things so that every layer along the path isn't storing its own journal. In fact, many of the optimizations possible with zfs and btrfs are due to the fact that you eliminate all those layers. Rich
Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On Tue, May 27, 2014 at 11:21 AM, J. Roeleveld jo...@antarean.org wrote: Does anyone know how these will handle (and perform) with a possible 300+ snapshots per filesystem (or volume, as I think it's called)? I can't speak for zfs. I had upwards of 1000 snapshots on my system before I stopped creating them hourly and and just started having issues with it. I wouldn't really say it is ready for prime time, but it is workable. Of course, you'll still want backups - a million snapshots does you no good if some bug wipes out your filesystem. For one of my ENOSPC incidents I ended up just wiping the entire filesystem and restoring from backup, though if I kept at it I'd probably have been able to fix it. Oh, one other tip if you use btrfs - be sure you have a rescue disk that supports it. Hint, the old Gentoo install CD I had lying around didn't. You'll probably want to keep a rescue CD with a recent kernel and btrfs-tools handy at all times. Rich
Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On Tue, 27 May 2014 11:32:22 -0400, Rich Freeman wrote: Oh, one other tip if you use btrfs - be sure you have a rescue disk that supports it. Hint, the old Gentoo install CD I had lying around didn't. You'll probably want to keep a rescue CD with a recent kernel and btrfs-tools handy at all times. I have a couple of System Rescue Cd ISOs in /boot and GRUB entries to boot them. I have two because I wanted the latest for btrfs but couldn't be bothered remastering for ZFS so i have an older, premastered image for that. -- Neil Bothwick Mr. bullfrog says: time's fun when you're having flies. signature.asc Description: PGP signature
Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]
On 27/05/2014 17:12, J. Roeleveld wrote: I have a yearly (full), monthly, weekly and daily. Each incremental is against the most recent one of itself or longer period. That means having to keep multiple snapshots active, which I prefer to avoid. But, it is a good idea for backing up desktops and laptops. I'm curious why you have yearly snapshots. I've yet to find any sane production system where a yearly backup had any worth at all. Even monthly is pushing it... Or do you do it to have a decent start point for incrementals? -- Alan McKinnon alan.mckin...@gmail.com
[gentoo-user] backups [was: Boot speedup]
On Tue, 13 Apr 2010 12:44:31 +0200 Alex Schuster wrote: Peter Humphrey writes: On Monday 12 April 2010 17:17:52 Florian Philipp wrote: Unless something is broken, I hardly ever reboot. How do you take backups? I do my backups from the running system, not from a live-cd. I create an LVM snapshot of the partition, and backup with use rdiff-backup. his way it does not matter if the partition itself is being modified during the backup. Wonko Backuppc works nicely here!
[gentoo-user] Backups...
Well, now that I've got my systems cleaned up, and KDE3 removed, I'm tackling another project I've been meaning to do - backups. Here's my basic plan: - I've got a directory on my server that I want to synchronize several systems with (some linux, and one Windows). - I want clients to push the backup; and not the server to pull it. - Clients may backup more than once a month. - the server will receive an additional backup itself once a month which includes all the client backups (may be more often, not sure). At least on the Linux Systems, I've settled to using rsync for the backup - easy enough to do. I'm already running an rsync server for hosting portage, so it's relatively trivial to add another rsync module to support that way, though I'm not sure what the best way is. rsync in attractive since it will do delta transfers to keep things in sync; though if I could use scp the same way I probably would since I would just have to setup appropriate keys. Any how...I setup the rsync daemon with a read-write section. Tested it, and it worked. But I'd really like to have it secured - I don't want anyone to be able to read/write to it. So I tried adding the following: [backup] uid = backup user gid = backup group path = /path/to/backup/repo read only = false list = false auth users = user secrets file = /path/to/rsyncd.secrets The rsyncd.secrets is simple: user:8 digit password If I don't have the last two lines (e.g. auth user, secrets file) then I can write to it. Otherwise I get an authentication error: @ERROR: auth failed on module backup rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] I'm uploading via: rsync -a --password-file=rsync.passwd someTestFile rsync://user@host/backup/extra/path/ rsync.passwd contains the same 8 digit password, nothing else. I've already checked file permissions - the entire directory structure under /path/to/backup/repo is owned by backup user:backup group. What am I doing wrong? Is there a better approach? TIA, Ben
Re: [gentoo-user] Backups
On Sunday 30 September 2007 01:45:41 Grant wrote: Do you back up anything other than /etc and /home on a standard system? - Grant /var because with /var gone its complete-reinstall time. What about splitting tar.gz files across multiple CDs? Can that be done? - Grant You might also want to have a look at dar, it will alow you to specify the size of every file (and even have a different size for the first file, usefull if you want to put a linux rescue install on the first cd) -- Joost -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Hi Grant, on Saturday, 2007-09-29 at 16:28:36, you wrote: Do you back up hidden files and directories in the home directory? There seems to be a lot of junk in there. Does something like '--exclude /home/user/.*' work with tar? It certainly does, but I'm quite sure it's not what you want. For me at least losing all my carefully customized stuff in .mutt, .gnupg, .bashrc, .vim etc. would suck asinine reproductive glands. It's usually all text anyway that compresses very well. cheers, Matthias -- I prefer encrypted and signed messages. KeyID: FAC37665 Fingerprint: 8C16 3F0A A6FC DF0D 19B0 8DEF 48D9 1700 FAC3 7665 pgpGkHKAShE1p.pgp Description: PGP signature
Re: [gentoo-user] Backups
On Sunday 30 September 2007, Volker Armin Hemmann wrote: On Sonntag, 30. September 2007, Grant wrote: Do you back up anything other than /etc and /home on a standard system? - Grant /var because with /var gone its complete-reinstall time. /usr/local too, otherwise you get to re-install everything in there as well alan -- Optimists say the glass is half full, Pessimists say the glass is half empty, Developers say wtf is the glass twice as big as it needs to be? Alan McKinnon alan at linuxholdings dot co dot za +27 82, double three seven, one nine three five -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Steen Eugen Poulsen schrieb: Grant skrev: Do you back up anything other than /etc and /home on a standard system? In a Gentoo system nothing is really standard, so I backup everything from / and then have a small exclude list with things like: /dev, /proc, /sys, /exports, /var/cache/squid, /srv/BackupPC. /var contains the most important files on a Gentoo system, if you loss the portage installed software information your in a heap of trouble. Goes both ways though backup of portages var information is less useful without backup of everything installed. I'd like to know which parts from /var are actually needed. Do we need anything more than /var/mail, /var/lib/portage/world and /var/log? /var/db seems to contain data about installed packages but since we don't back up /usr (when doing a minimal backup) we have to reemerge everything and these data should be regenerated automatically, don't you think? Minimal backup - if your willing to spend the hours on getting thing back up and running. /etc /root /home /usr/local /var I'd add /proc/config.gz to save the kernel config. -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Do you back up anything other than /etc and /home on a standard system? In a Gentoo system nothing is really standard, so I backup everything from / and then have a small exclude list with things like: /dev, /proc, /sys, /exports, /var/cache/squid, /srv/BackupPC. /var contains the most important files on a Gentoo system, if you loss the portage installed software information your in a heap of trouble. Goes both ways though backup of portages var information is less useful without backup of everything installed. I'm undecided about /usr/portage, I could save some time backing up /usr/portage/distfiles, but it is easily generated content. Minimal backup - if your willing to spend the hours on getting thing back up and running. /etc /root /home /usr/local /var For now I think I'll do /etc, /root, /home, /var/lib/portage/world, /usr/src/linux/.config, and anything specific I might need in /usr/local. What else am I missing out on in /var? I'm OK with a full reinstall for now. - Grant -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Do you back up anything other than /etc and /home on a standard system? - Grant /var because with /var gone its complete-reinstall time. What about splitting tar.gz files across multiple CDs? Can that be done? - Grant -- [EMAIL PROTECTED] mailing list A lot of things you're asking for can be accomplished with a script I've used (and successfully recovered with) called a stage 4 backup [1]. It's just your standard bash script that uses tar, gzip, bzip2, etc. to create manageable backups. I have my server set to backup once a month (I don't make significant changes to it very often). [1] http://gentoo-wiki.com/HOWTO_Custom_Stage4 That looks pretty slick. That will have to be the next step. - Grant -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Do you back up anything other than /etc and /home on a standard system? In a Gentoo system nothing is really standard, so I backup everything from / and then have a small exclude list with things like: /dev, /proc, /sys, /exports, /var/cache/squid, /srv/BackupPC. /var contains the most important files on a Gentoo system, if you loss the portage installed software information your in a heap of trouble. Goes both ways though backup of portages var information is less useful without backup of everything installed. I'm undecided about /usr/portage, I could save some time backing up /usr/portage/distfiles, but it is easily generated content. Minimal backup - if your willing to spend the hours on getting thing back up and running. /etc /root /home /usr/local /var For now I think I'll do /etc, /root, /home, /var/lib/portage/world, /usr/src/linux/.config, and anything specific I might need in /usr/local. What else am I missing out on in /var? I'm OK with a full reinstall for now. - Grant /boot/grub/grub.conf too. Does anyone leave /boot mounted all the time? - Grant -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Grant wrote: Do you back up anything other than /etc and /home on a standard system? In a Gentoo system nothing is really standard, so I backup everything from / and then have a small exclude list with things like: /dev, /proc, /sys, /exports, /var/cache/squid, /srv/BackupPC. /var contains the most important files on a Gentoo system, if you loss the portage installed software information your in a heap of trouble. Goes both ways though backup of portages var information is less useful without backup of everything installed. I'm undecided about /usr/portage, I could save some time backing up /usr/portage/distfiles, but it is easily generated content. Minimal backup - if your willing to spend the hours on getting thing back up and running. /etc /root /home /usr/local /var For now I think I'll do /etc, /root, /home, /var/lib/portage/world, /usr/src/linux/.config, and anything specific I might need in /usr/local. What else am I missing out on in /var? I'm OK with a full reinstall for now. - Grant /boot/grub/grub.conf too. Does anyone leave /boot mounted all the time? - Grant I do. I'm on dial-up so even if they can catch me online, they can't get anything big. It's to slow to upload to them and just as slow to send me something. 26K dial-up sucks but DSL is coming soon. Dale :-) :-)
Re: [gentoo-user] Backups
Do you back up anything other than /etc and /home on a standard system? In a Gentoo system nothing is really standard, so I backup everything from / and then have a small exclude list with things like: /dev, /proc, /sys, /exports, /var/cache/squid, /srv/BackupPC. /var contains the most important files on a Gentoo system, if you loss the portage installed software information your in a heap of trouble. Goes both ways though backup of portages var information is less useful without backup of everything installed. I'm undecided about /usr/portage, I could save some time backing up /usr/portage/distfiles, but it is easily generated content. Minimal backup - if your willing to spend the hours on getting thing back up and running. /etc /root /home /usr/local /var For now I think I'll do /etc, /root, /home, /var/lib/portage/world, /usr/src/linux/.config, and anything specific I might need in /usr/local. What else am I missing out on in /var? I'm OK with a full reinstall for now. - Grant Where do you guys store your backups? Leaving backups on a DVD in the same apartment as the machines doesn't make too much sense to me. Maybe I should mail em to my parents every week or something? - Grant -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Grant schrieb: Do you back up anything other than /etc and /home on a standard system? In a Gentoo system nothing is really standard, so I backup everything from / and then have a small exclude list with things like: /dev, /proc, /sys, /exports, /var/cache/squid, /srv/BackupPC. /var contains the most important files on a Gentoo system, if you loss the portage installed software information your in a heap of trouble. Goes both ways though backup of portages var information is less useful without backup of everything installed. I'm undecided about /usr/portage, I could save some time backing up /usr/portage/distfiles, but it is easily generated content. Minimal backup - if your willing to spend the hours on getting thing back up and running. /etc /root /home /usr/local /var For now I think I'll do /etc, /root, /home, /var/lib/portage/world, /usr/src/linux/.config, and anything specific I might need in /usr/local. What else am I missing out on in /var? I'm OK with a full reinstall for now. - Grant Where do you guys store your backups? Leaving backups on a DVD in the same apartment as the machines doesn't make too much sense to me. Maybe I should mail em to my parents every week or something? - Grant I keep them on an USB-stick (udf filesystem, with the same settings like a CD-RW). -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Do you back up anything other than /etc and /home on a standard system? In a Gentoo system nothing is really standard, so I backup everything from / and then have a small exclude list with things like: /dev, /proc, /sys, /exports, /var/cache/squid, /srv/BackupPC. /var contains the most important files on a Gentoo system, if you loss the portage installed software information your in a heap of trouble. Goes both ways though backup of portages var information is less useful without backup of everything installed. I'm undecided about /usr/portage, I could save some time backing up /usr/portage/distfiles, but it is easily generated content. Minimal backup - if your willing to spend the hours on getting thing back up and running. /etc /root /home /usr/local /var For now I think I'll do /etc, /root, /home, /var/lib/portage/world, /usr/src/linux/.config, and anything specific I might need in /usr/local. What else am I missing out on in /var? I'm OK with a full reinstall for now. - Grant Where do you guys store your backups? Leaving backups on a DVD in the same apartment as the machines doesn't make too much sense to me. Maybe I should mail em to my parents every week or something? - Grant I keep them on an USB-stick (udf filesystem, with the same settings like a CD-RW). But where do you put the USB stick? If my apartment building burns to the ground while I'm away, I'll lose my systems and the backups. - Grant -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
On 30 Sep 2007, at 12:33, Grant wrote: Where do you guys store your backups? Leaving backups on a DVD in the same apartment as the machines doesn't make too much sense to me. Maybe I should mail em to my parents every week or something? - Grant Offsite backups are a good idea if your data is important to you. I have several servers around the world so setting up rsync mirrors is pretty painless. Then I burn to DVD remotely.. (I have trained people enough so that when the tray opens they replace the DVD with a blank) You might want to look at doing a stage 4 backup and then sending that file to one of those online storage services. A quick google shows there are many out there offering 2-25GB of free storage. There are some non-free services designed specifically for backup where you don't pay to upload but do pay when you want to get your data (which given it is a backup I assume you would be highly motivated to pay if a restore is required) A few months back I looked into Amazon's S3 to automate offsite storage of backups. I never implemented anything though. I would encrypt anything sent to one of those online storage services. -- Christopher -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
On Sunday 30 September 2007, Grant wrote: I keep them on an USB-stick (udf filesystem, with the same settings like a CD-RW). But where do you put the USB stick? If my apartment building burns to the ground while I'm away, I'll lose my systems and the backups. I can't believe you actually asked that. Think, man, think. Here's some ways to start: Leave it at your mum's house where you have dinner every second day Leave it at your girlfriend's house Leave it in a safety box at the post office/your bank Leave it at a friend's house Leave it in the desk drawer at work Hang it off your keyring so it's always on your person Mail the data to your gmail account Upload the data to your off-site web/ftp/whatever server Store the backups in your car boot alan -- Optimists say the glass is half full, Pessimists say the glass is half empty, Developers say wtf is the glass twice as big as it needs to be? Alan McKinnon alan at linuxholdings dot co dot za +27 82, double three seven, one nine three five -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Where do you guys store your backups? Leaving backups on a DVD in the same apartment as the machines doesn't make too much sense to me. Maybe I should mail em to my parents every week or something? - Grant Offsite backups are a good idea if your data is important to you. I have several servers around the world so setting up rsync mirrors is pretty painless. Then I burn to DVD remotely.. (I have trained people enough so that when the tray opens they replace the DVD with a blank) You might want to look at doing a stage 4 backup and then sending that file to one of those online storage services. A quick google shows there are many out there offering 2-25GB of free storage. There are some non-free services designed specifically for backup where you don't pay to upload but do pay when you want to get your data (which given it is a backup I assume you would be highly motivated to pay if a restore is required) A few months back I looked into Amazon's S3 to automate offsite storage of backups. I never implemented anything though. I would encrypt anything sent to one of those online storage services. Encryption yeah. How do you encrypt your stuff? - Grant -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
I keep them on an USB-stick (udf filesystem, with the same settings like a CD-RW). But where do you put the USB stick? If my apartment building burns to the ground while I'm away, I'll lose my systems and the backups. I can't believe you actually asked that. Think, man, think. But hindsight is so much clearer than foresight. Good suggestions BTW. - Grant Here's some ways to start: Leave it at your mum's house where you have dinner every second day Leave it at your girlfriend's house Leave it in a safety box at the post office/your bank Leave it at a friend's house Leave it in the desk drawer at work Hang it off your keyring so it's always on your person Mail the data to your gmail account Upload the data to your off-site web/ftp/whatever server Store the backups in your car boot alan -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Hello Grant, Where do you guys store your backups? Leaving backups on a DVD in the same apartment as the machines doesn't make too much sense to me. Maybe I should mail em to my parents every week or something? I use rsync.net, offsite backups using duplicity for GPG encryption. -- Neil Bothwick Anything is good and useful if it's made of chocolate. signature.asc Description: PGP signature
Re: [gentoo-user] Backups
Hello Grant, For now I think I'll do /etc, /root, /home, /var/lib/portage/world, /usr/src/linux/.config, and anything specific I might need in /usr/local. What else am I missing out on in /var? Other data in /var/lib. For example, any databases kept in /var/lib/mysql. -- Neil Bothwick Every morning is the dawn of a new error... signature.asc Description: PGP signature
Re: [gentoo-user] Backups
On Sunday 30 September 2007 12:31:51 pm Grant wrote: Do you back up anything other than /etc and /home on a standard system? snip... Just my two cents worth here. Often I find a need to generate a duplicate of an existing gentoo installation and to ease the build process I run this script via cron... #!/bin/sh rm /portage.list/*.* emerge -pe --color=n system /portage.list/system.list emerge -pe --color=n world /portage.list/world.list Basicly it generates a list of installed ebuilds for both the system and world model. In my normal backup routines I add /portage.list... A great way to help rebuild an exact duplicate of an existing gentoo box. Cheers... -- From the Desk of: Jerome D. McBride -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
On Sunday 30 September 2007, Neil Bothwick wrote: Other data in /var/lib. For example, any databases kept in /var/lib/mysql. Rather than backup MySQL's or Postgres' binary storage I prefer to use the relevant tool (mysqldump, pgdump[all]) to backup the database to /root/backups/ just prior to running the real backup job. This has the advantage that you can generally restore onto any version of the database server, rather than the specific version that was running when the backup was taken. I'm not 100% sure but I thought this method of dumping the database is the only way to guarantee consistency of the snapshot without having to stop the database server for the duration of the backup. From what I understand when backing up a live database Postgres guarantees the consistency using MVCC but MySQL may be a little less careful if INSERTs/UPDATEs happen during the backup. For keeping a hot-spare failover database server closely in sync with realtime changes I'd suggest Slony1. Unless the database contains blobs I prefer to dump as plain text INSERTs and compress using gzip patched with --rsyncable support which leads me to... After backing up the database I /always/ backup to a remote machine, sometimes two, using rdiff-backup. It does point-in-time recovery and supports ACLs and xattr. The backup is not in some odd file format; it's stored in a subfolder of your filesystem just like any other tree of files so it is instantly available. A subfolder called rdiff-backup-data stores all the rollbacks and also all the acl/xattr/other metadata so the destination filesystem doesn't need to support them. There was even a GSoC project to provide a FUSE interface to mount the backup folder as it appeared at the time of some previous backup. This is all /very/ nice for when used in combination with Linux-VServer. With a little thought about XIDs you can start a backed-up VServer on your development box ;-) This is the essence of what I run in a cron job on the machine needing to be backed up: #!/bin/bash #/root/backup_mysql.sh #/root/backup_postgres.sh mount /boot /dev/null 2 /dev/null rdiff-backup --force --remove-older-than 90d \ [EMAIL PROTECTED]::~/backups/myhostname/ echo time rdiff-backup \ -v2 --print-statistics \ --exclude /mnt \ --exclude /media \ --exclude /dev \ --exclude /proc \ --exclude /tmp \ --exclude /var/tmp \ --exclude /var/cache/squid/ \ --exclude /var/cache/http-replicator/ \ --exclude /var/lib/mysql/ \ --exclude /var/lib/postgresql/data/base/ \ --exclude /var/lib/postgresql/data/global/ \ --exclude /var/lib/postgresql/data/pg_clog/ \ --exclude /var/lib/postgresql/data/pg_subtrans/ \ --exclude /var/lib/postgresql/data/pg_tblspc/ \ --exclude /var/lib/postgresql/data/pg_xlog/ \ --exclude /sys/ \ --exclude /usr/portage/ \ --exclude /usr/portage/distfiles \ / [EMAIL PROTECTED]::~/backups/myhostname/ umount /boot /dev/null 2 /dev/null -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Hello Jerry McBride, Just my two cents worth here. Often I find a need to generate a duplicate of an existing gentoo installation and to ease the build process I run this script via cron... #!/bin/sh rm /portage.list/*.* emerge -pe --color=n system /portage.list/system.list emerge -pe --color=n world /portage.list/world.list Basicly it generates a list of installed ebuilds for both the system and world model. Why not just backup the world list itself, /var/lib/portage/world? Your method doesn't distinguish between packages in world and their dependencies, emerging from this would result in a screwed world file. The system list is contained in your profile, so you just need a note of where /etc/make.profile points, which you will already have is you backup /etc. -- Neil Bothwick If you smoke after sex, you're doing it too fast. signature.asc Description: PGP signature
Re: [gentoo-user] Backups
On Sunday 30 September 2007 06:02:30 pm Neil Bothwick wrote: Hello Jerry McBride, Just my two cents worth here. Often I find a need to generate a duplicate of an existing gentoo installation and to ease the build process I run this script via cron... #!/bin/sh rm /portage.list/*.* emerge -pe --color=n system /portage.list/system.list emerge -pe --color=n world /portage.list/world.list Basicly it generates a list of installed ebuilds for both the system and world model. Why not just backup the world list itself, /var/lib/portage/world? Your method doesn't distinguish between packages in world and their dependencies, emerging from this would result in a screwed world file. It doesn't have too. The files I listed plus a backup of /etc is all you need... The system list is contained in your profile, so you just need a note of where /etc/make.profile points, which you will already have is you backup /etc. -- From the Desk of: Jerome D. McBride -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Hello Jerry McBride, Why not just backup the world list itself, /var/lib/portage/world? Your method doesn't distinguish between packages in world and their dependencies, emerging from this would result in a screwed world file. It doesn't have too. The files I listed plus a backup of /etc is all you need... So how do you create your world file? And why do you need a list of system files when you have already backed up /etc? -- Neil Bothwick Top Oxymorons Number 37: Sanitary landfill signature.asc Description: PGP signature
Re: [gentoo-user] Backups
On Sat, 29 Sep 2007 21:35:42 -0400 Kenneth Prugh [EMAIL PROTECTED] wrote: On Sat, 29 Sep 2007 16:28:36 -0700 Grant [EMAIL PROTECTED] wrote: Does something like '--exclude /home/user/.*' work with tar? - Grant Yes you may exclude files from being included. From the tar man page: --exclude PATTERN exclude files based upon PATTERN -X, --exclude-from FILE exclude files listed in FILE However, don't exclude files you want, such as .bash*, .xsession, and so on. Generally these files aren't large and are important to the private storage of uesr-specific info for many programs. -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
On Sun, 30 Sep 2007 20:15:04 +0100 Neil Bothwick [EMAIL PROTECTED] wrote: Hello Grant, For now I think I'll do /etc, /root, /home, /var/lib/portage/world, /usr/src/linux/.config, and anything specific I might need in /usr/local. What else am I missing out on in /var? Other data in /var/lib. For example, any databases kept in /var/lib/mysql. BIND, Apache, various mailing programs, at, Xauth, logs if you want to know what happened on a particular computer six months from now (hard to foresee this kind of thing). I think you can clear out /var/tmp if you wish. Also many of the logs can be cleared and such. After clearing all the data you don't need (be careful; some data is important to portage), you shouldn't have too much more than a few hundred megs, pretty much all highly compressable text. -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
On Sat, 29 Sep 2007 15:26:07 -0700 Grant [EMAIL PROTECTED] wrote: Do you back up anything other than /etc and /home on a standard system? - Grant Don't forget to back up stuff that can help you rebuild the system quickly. Like /proc/config.gz, or better yet just the kernel and modules you need so you don't have to rebuild at all or generate the sources. Another thing that I think is highly valuable to back up, and very often ignored, is the output of 'fdisk -l'. If your drive dies it's very nice to have a reminder of how it was formatted. -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Dan Farrell wrote: On Sat, 29 Sep 2007 15:26:07 -0700 Grant [EMAIL PROTECTED] wrote: Do you back up anything other than /etc and /home on a standard system? - Grant Don't forget to back up stuff that can help you rebuild the system quickly. Like /proc/config.gz, or better yet just the kernel and modules you need so you don't have to rebuild at all or generate the sources. Another thing that I think is highly valuable to back up, and very often ignored, is the output of 'fdisk -l'. If your drive dies it's very nice to have a reminder of how it was formatted. In order to be able to restore a system (relatively) quickly, I use the appropriate fs dump tool (xfsdump in my case) to make level 0 backups of /boot, / , /usr, /var after a major configuration change (e.g emerge --sync;emerge -u world), along with output from df -m. This does not take too long (/usr does take a while), but really speeds up a restore (I have sufficient packages installed to make an emerge world take 10 hours). For a modern server with minimal software actually installed, the time aspect for this method may not be too different from an install from scratch, but it also guarantees that the restored system is the same as it was before (modulo last backup obviously), which can save a lot of time! Cheers Mark P.s : Actually rebuilding from these saved dumps requires a little thought - I'll post the steps if anyone new to dumps is interested in using this method for themselves. -- [EMAIL PROTECTED] mailing list
[gentoo-user] Backups
Do you back up anything other than /etc and /home on a standard system? - Grant -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
On Sonntag, 30. September 2007, Grant wrote: Do you back up anything other than /etc and /home on a standard system? - Grant /var because with /var gone its complete-reinstall time. -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Do you back up anything other than /etc and /home on a standard system? - Grant /var because with /var gone its complete-reinstall time. Do you back up hidden files and directories in the home directory? There seems to be a lot of junk in there. Does something like '--exclude /home/user/.*' work with tar? - Grant -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Grant skrev: Do you back up anything other than /etc and /home on a standard system? In a Gentoo system nothing is really standard, so I backup everything from / and then have a small exclude list with things like: /dev, /proc, /sys, /exports, /var/cache/squid, /srv/BackupPC. /var contains the most important files on a Gentoo system, if you loss the portage installed software information your in a heap of trouble. Goes both ways though backup of portages var information is less useful without backup of everything installed. I'm undecided about /usr/portage, I could save some time backing up /usr/portage/distfiles, but it is easily generated content. Minimal backup - if your willing to spend the hours on getting thing back up and running. /etc /root /home /usr/local /var It's possible to make an exclude list for /var as it contains both variable data thats important to save and dynamic data that will be generated again. smime.p7s Description: S/MIME Cryptographic Signature
Re: [gentoo-user] Backups
On Sonntag, 30. September 2007, Grant wrote: Do you back up anything other than /etc and /home on a standard system? - Grant /var because with /var gone its complete-reinstall time. What about splitting tar.gz files across multiple CDs? Can that be done? - Grant man tar: -M, --multi-volume -L, --tape-length= -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
On Sonntag, 30. September 2007, Grant wrote: Do you back up anything other than /etc and /home on a standard system? - Grant /var because with /var gone its complete-reinstall time. Do you back up hidden files and directories in the home directory? yes, they contain settings, pw, all the juicy stuff. I tar everything onto tape. tar -c -b 128 /home/energyman | mbuffer -m 400M -p 95 -s 65536 /dev/st0 There seems to be a lot of junk in there. Does something like '--exclude /home/user/.*' work with tar? don't know, never tried. -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
On Sat, 2007-09-29 at 15:26 -0700, Grant wrote: Do you back up anything other than /etc and /home on a standard system? I pretty much back up everything that's not ubiquitous on the Internet on a on an external drive. Disk space is so cheap these days so I figure why not. -- Albert W. Hopkins -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
Do you back up anything other than /etc and /home on a standard system? - Grant /var because with /var gone its complete-reinstall time. What about splitting tar.gz files across multiple CDs? Can that be done? - Grant -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Backups
On 9/29/07, Grant [EMAIL PROTECTED] wrote: Do you back up anything other than /etc and /home on a standard system? - Grant /var because with /var gone its complete-reinstall time. What about splitting tar.gz files across multiple CDs? Can that be done? - Grant -- [EMAIL PROTECTED] mailing list A lot of things you're asking for can be accomplished with a script I've used (and successfully recovered with) called a stage 4 backup [1]. It's just your standard bash script that uses tar, gzip, bzip2, etc. to create manageable backups. I have my server set to backup once a month (I don't make significant changes to it very often). [1] http://gentoo-wiki.com/HOWTO_Custom_Stage4 -- - Mark Shields
[gentoo-user] Backups... a very general question...
I've recently been thinking about backup strategy... following a painful re-install after dropping a clanger during a kernel upgrade. While this seems a very basic topic, I can find surprisingly little documentation about this on-line. I need to address several entirely different kinds of backup: 1. A backup of my root boot partitions in a working state. This should be a backup to DVD-RW(s) - and would not contain any user-data... but would provide a recovery point to get a working server as quickly as possible in the event of a drive failure. It would be fantastic if, in addition to this. there were some means to track which packages had been merged/updated since the backup was made - and a copy to be made of any configuration changes... The list of updated packages (and the versions to which they've been updated) and any changes to configuration files would be tiny and hence easy to backup via another approach. It would be fantastic if the backup DVDs were bootable and doing so would restore the backup. 2. I've many gigabytes of MP3 files stored in Artist/[year]Album/*.* hierarchy... which I extend sporadically. I'd like a backup of this (as organising it took lots of time) but a different approach is necessary here... I'd like to pack as many whole albums onto DVDRs as would fit, which I'd then number, and given a list detailing which albums are on which DVDs, I could also play albums from a DVD player attached to a hi-fi. I'd like to be prompted to backup each time N-Mb of new data has been added to my MP3 directory - and that the most recent DVD-R should be authored with minimum user intervention. 3. My home directory; subversion repositories and DBMS catalogues are backed-up to a remote account. I currently do this with a cron-job which takes dumps; creates tar files; AES encrypts then uploads using SSH to the remote site... which manages a history of 3 backups using a simple shell-script. This works OK, but it is very ad-hoc... and it won't scale as every backup requires that I upload a new copy - even if I've only made a trivial change to my data. It would be far better if an incremental update were possible - though I'm not willing to give up encryption of data I send off-site. Are there any packages which would make any (or all) of these tasks more straightforward or more efficient? -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Backups... a very general question...
On Wed, 27 Sep 2006 14:04:26 +0100, Steve [Gentoo] wrote: 3. My home directory; subversion repositories and DBMS catalogues are backed-up to a remote account. I currently do this with a cron-job which takes dumps; creates tar files; AES encrypts then uploads using SSH to the remote site... which manages a history of 3 backups using a simple shell-script. This works OK, but it is very ad-hoc... and it won't scale as every backup requires that I upload a new copy - even if I've only made a trivial change to my data. It would be far better if an incremental update were possible - though I'm not willing to give up encryption of data I send off-site. Does your remote site use rsync? I use Strongspace and have a directory on the server set up with encfs. I tried mounting it with sshfs and then encfs but found it very slow, so what I now do is have a local directory, mounted with encfs. I backup from my home directory to that, using rsync, then I rsync that encrypted directory with the one on the server, so I am transferring pre-encrypted files. I can still mount the remote directory using sshfs and encfs if I need to, and if I need to, the lack of speed won't be my main concern. -- Neil Bothwick Bother said Rue, for no apparent reason signature.asc Description: PGP signature
RE: [gentoo-user] Backups
-Original Message- From: Neil Bothwick [mailto:[EMAIL PROTECTED] Sent: 15 December 2005 19:16 To: gentoo-user@lists.gentoo.org Subject: Re: [gentoo-user] Backups On Thu, 15 Dec 2005 09:53:27 -0700, Richard Fish wrote: My personal favorite for my desktop and laptop is using 'dar' with big USB hard drivesbut that's what works well for me. I use rdiff-backup, which is ideal for backing up automatically to a hard drive. I run it from cron, hourly on critical directories, daily on the rest. I then have a weekly cron script that compresses the backup directories with squashfs and writes them to ISO images ready for writing to bootable DVDs. It makes restoring individual files very easy, and a completely hosed system can be fixed because the DVDs are bootable. If you get a minute, a detailed wiki howto would be useful for some of us. :-) -- Regards, Mick -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Backups
On Thu, 22 Dec 2005 14:26:56 -, Michael Kintzios wrote: If you get a minute, a detailed wiki howto would be useful for some of us. :-) A minute, where can I get one of those? Is it in portage? :-( -- Neil Bothwick Eagles may soar, but Wombles don't get sucked into jet engines signature.asc Description: PGP signature
Re: [OT] how to make Bootable DVD/CDs [Was] Re: [gentoo-user] Backups
On 16 December 2005 08:11, Ow Mun Heng wrote: On Thu, 2005-12-15 at 23:02 -0700, Richard Fish wrote: On 12/15/05, Ow Mun Heng [EMAIL PROTECTED] wrote: -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4 -boot-info-table Thanks. Wonderful info. The grub info pages contain a whole page about writing bootable CDs and DVDs. Uwe -- Unix is sexy: who | grep -i blonde | date cd ~; unzip; touch; strip; finger mount; gasp; yes; uptime; umount sleep -- gentoo-user@gentoo.org mailing list
Re: [OT] how to make Bootable DVD/CDs [Was] Re: [gentoo-user] Backups
On Fri, 16 Dec 2005 11:40:32 +0800, Ow Mun Heng wrote: how can one create bootable CDs/DVDs? Is there a simple way to transfer GRUB into the DVD/CDs? Or would dd of the /boot partition transfer the whole thing?? I created a rescue system, containing all the tools I need by following this guide: http://forums.gentoo.org/viewtopic-t-244837.html -- Neil Bothwick Runtime Error: Out of funny taglines! signature.asc Description: PGP signature
Re: [gentoo-user] Backups
Allan Spagnol Comar wrote: Thanks all for the answers until now; What i am looking for is to backup 3 servers, and a critical issue workstation. I think that you should test Bacula. that is very scalable app to backup from different sources to different media. http://www.bacula.org Greets Pawel -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Backups
thanks. On 12/16/05, Paweł Madej [EMAIL PROTECTED] wrote: Allan Spagnol Comar wrote: Thanks all for the answers until now; What i am looking for is to backup 3 servers, and a critical issue workstation. I think that you should test Bacula. that is very scalable app to backup from different sources to different media. http://www.bacula.org Greets Pawel -- gentoo-user@gentoo.org mailing list -- An application asked: Requeires Windows 9x, NT4 or better, so I´ve installed Linux -- gentoo-user@gentoo.org mailing list
[gentoo-user] Backups
Good day Gentoo List !!! Some one knows where I can find some good material about backuping linux boxes ? thanks, Allan -- An application asked: Requeires Windows 9x, NT4 or better, so I´ve installed Linux -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Backups
There is a book covering backups that is supposed to be the Bible for backups. Unfortunately, I am not where I can lay hands on it and can't remember the title. From: Allan Spagnol Comar [EMAIL PROTECTED] Date: 2005/12/15 Thu AM 08:03:34 EST To: gentoo-user@lists.gentoo.org Subject: [gentoo-user] Backups Good day Gentoo List !!! Some one knows where I can find some good material about backuping linux boxes ? thanks, Allan -- An application asked: Requeires Windows 9x, NT4 or better, so I´ve installed Linux -- gentoo-user@gentoo.org mailing list -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Backups
On 12/15/05, Allan Spagnol Comar [EMAIL PROTECTED] wrote: Some one knows where I can find some good material about backuping linux boxes ? If you can give us some more details about what you are looking for (number of boxes, backup device you want to use, whether you want simple backup or archiving, etc), you can probably get some better answers. My personal favorite for my desktop and laptop is using 'dar' with big USB hard drivesbut that's what works well for me. -Richard -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Backups
On Thu, Dec 15, 2005 at 09:53:27AM -0700, Richard Fish wrote: My personal favorite for my desktop and laptop is using 'dar' with big USB hard drivesbut that's what works well for me. ditto - very easy, very efficient John pgpdB8qaDSaQE.pgp Description: PGP signature
Re: [gentoo-user] Backups
On Thu, 15 Dec 2005 09:53:27 -0700, Richard Fish wrote: My personal favorite for my desktop and laptop is using 'dar' with big USB hard drivesbut that's what works well for me. I use rdiff-backup, which is ideal for backing up automatically to a hard drive. I run it from cron, hourly on critical directories, daily on the rest. I then have a weekly cron script that compresses the backup directories with squashfs and writes them to ISO images ready for writing to bootable DVDs. It makes restoring individual files very easy, and a completely hosed system can be fixed because the DVDs are bootable. -- Neil Bothwick A day without sunshine is like night. signature.asc Description: PGP signature
Re: [gentoo-user] Backups
On 12/15/05, Neil Bothwick [EMAIL PROTECTED] wrote: completely hosed system can be fixed because the DVDs are bootable. So are my USB hard disks. Nothing against DVD backups though...I just find them too slow and small for my needs... -Richard -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Backups
Thanks all for the answers until now; What i am looking for is to backup 3 servers, and a critical issue workstation. I have a storage working with samba, so my bakups will go to this samba server. I would like to make some diff bakups to save storage space :) thanks again all of you got interested on backups on large USB disks too ... On 12/15/05, Richard Fish [EMAIL PROTECTED] wrote: On 12/15/05, Neil Bothwick [EMAIL PROTECTED] wrote: completely hosed system can be fixed because the DVDs are bootable. So are my USB hard disks. Nothing against DVD backups though...I just find them too slow and small for my needs... -Richard -- gentoo-user@gentoo.org mailing list -- An application asked: Requeires Windows 9x, NT4 or better, so I´ve installed Linux -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Backups
On Thu, 15 Dec 2005 13:04:38 -0700, Richard Fish wrote: completely hosed system can be fixed because the DVDs are bootable. So are my USB hard disks. Nothing against DVD backups though...I just find them too slow and small for my needs... A fair point, although it does make keeping spare, off-site backups rather expensive :( rdiff-backup should be excellent with a USB hard disk, I may give it a go. -- Neil Bothwick Jimi Hendrix's modem was a Purple Hayes. signature.asc Description: PGP signature
[OT] how to make Bootable DVD/CDs [Was] Re: [gentoo-user] Backups
On Thu, 2005-12-15 at 19:15 +, Neil Bothwick wrote: the rest. I then have a weekly cron script that compresses the backup directories with squashfs and writes them to ISO images ready for writing to bootable DVDs. It makes restoring individual files very easy, and a completely hosed system can be fixed because the DVDs are bootable. how can one create bootable CDs/DVDs? Is there a simple way to transfer GRUB into the DVD/CDs? Or would dd of the /boot partition transfer the whole thing?? -- Ow Mun Heng Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM 98% Microsoft(tm) Free!! Neuromancer 11:40:31 up 1 day, 20:17, 5 users, load average: 0.28, 0.36, 0.45 -- gentoo-user@gentoo.org mailing list
Re: [OT] how to make Bootable DVD/CDs [Was] Re: [gentoo-user] Backups
On 12/15/05, Ow Mun Heng [EMAIL PROTECTED] wrote: On Thu, 2005-12-15 at 19:15 +, Neil Bothwick wrote: the rest. I then have a weekly cron script that compresses the backup directories with squashfs and writes them to ISO images ready for writing to bootable DVDs. It makes restoring individual files very easy, and a completely hosed system can be fixed because the DVDs are bootable. how can one create bootable CDs/DVDs? Is there a simple way to transfer GRUB into the DVD/CDs? Or would dd of the /boot partition transfer the whole thing?? The boot process for a CD is very very different than from a hard drive. Basically the BIOS has to pretend that the CD drive is actually a floppy, or a hard drive, for implementing the BIOS calls used by the boot loader. Doing a dd of a boot partition and writing that to a CD would not work. There are several different software packages that can make bootable CDs, including GRUB. Ok, it isn't really GRUB that makes the CD bootable, but mkisofs. To use GRUB, you have to make a /boot/grub directory in your CD tree, and copy the stage2_eltorito file into that directory along with grub.conf/menu.lst. Then when you run mkisofs, you add the options: -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4 -boot-info-table Someone who uses a GUI for CD burning will have to comment on how to do this without the command line! -Richard -- gentoo-user@gentoo.org mailing list
Re: [OT] how to make Bootable DVD/CDs [Was] Re: [gentoo-user] Backups
On Thu, 2005-12-15 at 23:02 -0700, Richard Fish wrote: On 12/15/05, Ow Mun Heng [EMAIL PROTECTED] wrote: -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4 -boot-info-table Thanks. Wonderful info. /me just bought a DVD writer. -- Ow Mun Heng Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM 98% Microsoft(tm) Free!! Neuromancer 14:11:02 up 1 day, 22:47, 6 users, load average: 0.09, 0.17, 0.35 -- gentoo-user@gentoo.org mailing list
Re: [OT] how to make Bootable DVD/CDs [Was] Re: [gentoo-user] Backups
On 12/15/05, Richard Fish [EMAIL PROTECTED] wrote: bootable, but mkisofs. To use GRUB, you have to make a /boot/grub directory in your CD tree, and copy the stage2_eltorito file into that directory along with grub.conf/menu.lst. Oh, I forgot one thing. Instead of (hdX,X) in grub.conf, you use (cd). For example: title 2.6 kernel (cd)/boot/vmlinuz-2.6 root=/dev/hdc Of course, your kernel must have support for your CD-ROM drive and the iso9660 filesystem built in... -Richard -- gentoo-user@gentoo.org mailing list
