Re: Best procedure for full backup of live system
Nerius Landys wrote: dd if=/dev/ad4 of=MBR_backup bs=512 count=1 to back up the MBR, so I can recontruct the boot program and partition table. But they don't mention that in the Handbook. While that will back up the Master Boot Record, it's not sufficient for the general case of MS-DOS style partitions. It will only include the primary and extended partitions. The logical partitions, which exist inside an extended partition, are a linked list whose elements are distributed across your disk somewhere. If you have any extended partitions, you need to back them up too. They aren't part of the 512-byte MBR so just using dd to grab the first 512 bytes won't do. The Linux sfdisk program can create a machine readible text file that includes any logical partitions. Does FreeBSD also have sfdisk? Perhaps it's in ports. I don't have my FreeBSD box handy right now so I can't just look. ... and as has been pointed out, you'll also want your BSD disklabel which is a whole different beast. Mike -- Michael David Crawford m...@prgmr.com prgmr.com - We Don't Assume You Are Stupid. Xen-Powered Virtual Private Servers: http://prgmr.com/xen ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Best procedure for full backup of live system
Thanks again guys. My final series of steps to take full backups: bsdlabel ad4s1| ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/bsdlabel_ad4s1 dmesg -a | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dmesg dd if=/dev/ad4 bs=512 count=1 | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/MBR cat /etc/fstab| ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/fstab dump -0Lan -f - / | gzip | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dump0-root.gz dump -0Lan -f - /tmp | gzip | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dump0-tmp.gz dump -0Lan -f - /var | gzip | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dump0-var.gz dump -0Lan -f - /usr | gzip | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dump0-usr.gz ... where port 2 on localhost is a pipe to my remote desktop with the 500 GB harddrive. If I missed anything important please let me know. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Best procedure for full backup of live system
Warren Block wrote: Lastly, it says save all boot messages. Do they mean the output of dmesg? Why is this useful? It would show what hardware was in the old machine and what services are running on startup. Backup in general is making copies of information you won't need as long as you have a backup of it. The system already creates a copy of the first part of the system boot output for you: /var/run/dmesg.boot -- this shows you what hardware the kernel discovered as it probed all of the buses and so forth. It doesn't contain the rest of the output from all of the /etc/rc.d/* and /usr/local/etc/rc.d/* startup scripts. The simple way to capture all that is to enable console.log in /etc/syslog.conf -- you want to uncomment this line: console.info/var/log/console.log and make sure that /var/log/console.log exists, then restart syslog. This will log all console output, including the boot-up output. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
Re: Best procedure for full backup of live system
On Fri, 16 Oct 2009, Nerius Landys wrote: Thanks again guys. My final series of steps to take full backups: bsdlabel ad4s1| ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/bsdlabel_ad4s1 dmesg -a | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dmesg dd if=/dev/ad4 bs=512 count=1 | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/MBR cat /etc/fstab| ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/fstab dump -0Lan -f - / | gzip | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dump0-root.gz dump -0Lan -f - /tmp | gzip | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dump0-tmp.gz dump -0Lan -f - /var | gzip | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dump0-var.gz dump -0Lan -f - /usr | gzip | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dump0-usr.gz ... where port 2 on localhost is a pipe to my remote desktop with the 500 GB harddrive. If I missed anything important please let me know. The dump man page recommends always using -C with snapshot dumps. The Handbook has an example of dump over ssh that shows a couple of speed optimizations: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/backup-basics.html#AEN25817 gzip -2 and bluefish instead of 3des should be faster than the default. -Warren Block * Rapid City, South Dakota USA___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Best procedure for full backup of live system
On Fri, Oct 16, 2009 at 12:12:56AM -0700, Nerius Landys wrote: You can do all this though it might be more than needed. Only the level 0 dumps are needed. Thanks again guys. My final series of steps to take full backups: bsdlabel ad4s1| ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/bsdlabel_ad4s1 dmesg -a | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dmesg dd if=/dev/ad4 bs=512 count=1 | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/MBR cat /etc/fstab| ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/fstab dump -0Lan -f - / | gzip | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dump0-root.gz dump -0Lan -f - /tmp | gzip | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dump0-tmp.gz dump -0Lan -f - /var | gzip | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dump0-var.gz dump -0Lan -f - /usr | gzip | ssh -p 2 nlan...@localhost dd of=/home/nlandys/backup/dump0-usr.gz ... where port 2 on localhost is a pipe to my remote desktop with the 500 GB harddrive. If I missed anything important please let me know. Are you clear about what you have to run on the other machine to receive the data and put it where you want? jerry ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Best procedure for full backup of live system
On Thu, 15 Oct 2009 21:47:57 -0600 (MDT), Warren Block wbl...@wonkity.com wrote: Just a general note: backup to a hard drive isn't bad, but it's not the same as removable media. One failure can kill all of your backups... That's why it's often a good choice two have at least two hard disks (maybe external ones) for backup, so if one fails (which seems to be a problem of modern disks rather than older ones), there's still an intact backup on the other one (or on one of the others). Backup in general is making copies of information you won't need as long as you have a backup of it.a A wise summary. :-) dump(8) doesn't do all sectors, just ones used by the filesystem. Also, dump doesn't cross filesystems. In a typical FreeBSD install, /, /var, and /usr are separate filesystems. A dump of / won't get them all at once. The dump utility is good when you want to work partition-wise. If you have a setting where everything goes into a big /, dumping it will get all data - from that partition. Slices and MBR are out of dump's scope. My server should boot fine with the FreeBSD CDROM (fixit), because it uses a subset of the GENERIC kernel device drivers. If you can, try that before an actual emergency. Furthermore, it's good to check backups regularly. A defective backup is NO backup. If data doesn't restore as intended (e. g. to a testing system), then...? A situation that many of you surely have come across, as I have: Operator: The hard disk crashed, we need to restore from backups. Customer: Of course I have backups! Here! Customer hands over three tapes. Operator starts restore with tape #1. Operator: First tape is through. Good. Next one. Computer displays rroor reading /dev/nsa0: Tape is defect, cannot read. Operator: Do you have other set of tapes? This #2 is defective. Customer: Yes! Tape #3! Operator: I need a working tape #2. Customer: BUT I *HAVE* BACKUPS!!! Testing the backups may take some time, I agree, but it's mostly worth it - it's worth as much as your data is to you. -- Polytropon Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Best procedure for full backup of live system
Nerius Landys wrote: My server is increasingly having important work stored on it, and I need to start taking backups of a lot of directories, especially /home, /opt, /etc, /usr/local/etc, and maybe others. The ideal backup (and what I've done in the past) is to take a full low-level dd image of the disk while the system is down (this is easy to do in a situation where you have dual boot). ...but you've found recently that it isn't scalable. Or, since the output of dd would take up tons of space and would only be usable on an identical hard drive, use dump to take the backup while the machine is turned off (again easy to do on a dual boot). But now, I cannot bring down the machine. My plan is to do a tar gzip of / on the fly, and pipe that to ssh (remote machine). However, the system is live, and files will be in the progress of changing. ...quick hack, tested example ( recommended for a quick fix only. do a proper archive ) %pwd /home/steve %mkdir arch ls | grep arch arch %echo blah, blah arch/file.txt %cat !$ cat arch/file.txt blah, blah # now, even though this is a ~ example, I use / all the time... tarball /home/steve/arch directory ( and contents ) to a different server: %tar -cvzf - arch | ssh st...@amanda.eagle.ca ' cat arch_test.tar.gz' a arch a arch/file.txt ... after key-auth ( or password auth ) to remote SSH server, on the remote server: %pwd /usr/home/steve %ll | grep arch -rw-r--r-- 1 steve steve 10240 Oct 15 02:16 arch_test.tar.gz %pwd /usr/home/steve %mkdir restore-test %mv arch_test.tar.gz restore-test/ %cd !$ cd restore-test/ %tar -xzvf arch_test.tar.gz x arch/ x arch/file.txt %cd arch %ll total 1 -rw-r--r-- 1 steve steve 11 Oct 15 02:17 file.txt ... booya, restored, on a remote server...look: %cat file.txt blah, blah ...Just like that! If I've missed something, forgive me. I swear that doing a 'backup' to a remote location with a FreeBSD box is honestly *nearly* as easy to this Canadian as packing snow into an iceball to hit the bus with ;) Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Best procedure for full backup of live system
On Wed, Oct 14, 2009 at 08:42:47PM -0700, Nerius Landys wrote: My server is increasingly having important work stored on it, and I need to start taking backups of a lot of directories, especially /home, /opt, /etc, /usr/local/etc, and maybe others. The ideal backup (and what I've done in the past) is to take a full low-level dd image of the disk while the system is down (this is easy to do in a situation where you have dual boot). Or, since the output of dd would take up tons of space and would only be usable on an identical hard drive, use dump to take the backup while the machine is turned off (again easy to do on a dual boot). But now, I cannot bring down the machine. My plan is to do a tar gzip of / on the fly, and pipe that to ssh (remote machine). However, the system is live, and files will be in the progress of changing. My question is, what is the recommended procedure of taking a full backup on a live system? Ideally, if my hard drive were to crash, I would like to have such a backup so as to make it possible to copy over the entire backup to a new identical harddrive without doing any reinstall or configuration. Should I use tar/gzip? dump? What exact command should I use? I guess I'll back up all of / including system files, because there is not too much data. I will be piping the output to ssh. Use dump(8) to back up each filesystem that is important and that cannot be easily recreated (such as by reinstalling). Don't bother with any of that tar and dd stuff as long as the dump will be read on a similar system (FreeBSD). Use the -L switch for making a dump on a live filesystem. It forces a snapshot so files are not in transition while the backup is done - or rather, makes it so the backup is of an intact image. Your big issue then is where to write the dump, how often to do it and how many copies you want to keep of it. You can do full dumps and dumps of just what has changed since the last time a file was dumped. I call those full dumps and change dumps. The documentation referrs to them as level 0 for full dump and level 1-9 as the change dumps. The man pages give a complicated scheme for managing full and change dumps. Probably most people really need only a level 0 and a level 1, maybe a level 2. Basically the point of the change dumps is to make smaller backup images which takes less time and less media. You only make the full dump (level 0) once every week or every month - whatever your needs are. Then, in between you only dump the files that have changed since the last full dump. If that change dump file gets too big as well, then you jump to the next level on change dump. So, you do a level 0, then, the next day a level 1. If it is small (meaning only a relatively few files have changed) then the third day you still make a level 1. If the level 1 dump is now real big (meaning a lot of files changed) then on day 4 you go to a level 2 dump, etc. It is probably a good idea to regularize the process of choosing levels. That is why the man page has such a complicated scheme that covers all conditions. But, as I indicated, most people with a personal or office/department level server often need only a need the regular full (level 0) dump, plus a daily level 1 dump in between the full dumps.In fact, I have some servers that are small enough that I just make level 0 dumps each time. Now, if you have a big system with lots of new files and changed files all the time, then you will have to organize your dumps in a more sophisticated manner. Generally, level 0 dumps take whatever amount of media they need to contain the whole filesystem. Then, for the change dumps (level 1..9) you hope to keep then to only one unit of media. If a change dump goes over one unit of media, then you move up a level the next time. The same goes for if the change dump starts to take a lot of extra time. As for media, it can be to an external disk, a tape or over the net to some big storage space. Try to spread it out so that each set of dumps is not on the same physical media as other ones - eg rotate your media. jerry ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Best procedure for full backup of live system
Thanks for your help guys. I have decided to attempt the following. With a filesystem snapshot, take a dump 0 of all filesystems. Back up these dumps to a 500GB disk I have sitting at home (the server I'm backing up is in a data center). Perform this maybe once a week or once a month. I am now looking at this: http://www.freebsd.org/doc/en/books/handbook/backup-basics.html#AEN25994 For step 1, I'm a little unsure what they are talking about. I assume that to begin with I would do this: bsdlabel ad4s1 Since that is my disk and it has one partition. I would probably also want to do this: dd if=/dev/ad4 of=MBR_backup bs=512 count=1 to back up the MBR, so I can recontruct the boot program and partition table. But they don't mention that in the Handbook. Then it says to back up /etc/fstab, but I'm not quite sure how I'd use this in a restore. Lastly, it says save all boot messages. Do they mean the output of dmesg? Why is this useful? Of course I would complement all of these things with the actual dump 0 of all sectors on the primary partition of the single hard drive that I have. My server should boot fine with the FreeBSD CDROM (fixit), because it uses a subset of the GENERIC kernel device drivers. So I probably don't need to pay attention to the instuctions talking about floppies. Am I correct in all of this? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Best procedure for full backup of live system
On Thu, 15 Oct 2009, Nerius Landys wrote: Thanks for your help guys. I have decided to attempt the following. With a filesystem snapshot, take a dump 0 of all filesystems. Back up these dumps to a 500GB disk I have sitting at home (the server I'm backing up is in a data center). Perform this maybe once a week or once a month. Just a general note: backup to a hard drive isn't bad, but it's not the same as removable media. One failure can kill all of your backups... I am now looking at this: http://www.freebsd.org/doc/en/books/handbook/backup-basics.html#AEN25994 For step 1, I'm a little unsure what they are talking about. I assume that to begin with I would do this: bsdlabel ad4s1 Since that is my disk and it has one partition. The first step there has you print out the bsdlabel. That can be valuable because it's a lot easier to have that information on hand than to try to reconstruct it after a failure. I would probably also want to do this: dd if=/dev/ad4 of=MBR_backup bs=512 count=1 Handy to have, and takes little space. to back up the MBR, so I can recontruct the boot program and partition table. But they don't mention that in the Handbook. Then it says to back up /etc/fstab, but I'm not quite sure how I'd use this in a restore. Mostly it's reference information you might need after a failure, to see which partition goes to which mountpoint: Aha! On that drive, /usr is s1g, not s1f. Lastly, it says save all boot messages. Do they mean the output of dmesg? Why is this useful? It would show what hardware was in the old machine and what services are running on startup. Backup in general is making copies of information you won't need as long as you have a backup of it. Of course I would complement all of these things with the actual dump 0 of all sectors on the primary partition of the single hard drive that I have. dump(8) doesn't do all sectors, just ones used by the filesystem. Also, dump doesn't cross filesystems. In a typical FreeBSD install, /, /var, and /usr are separate filesystems. A dump of / won't get them all at once. My server should boot fine with the FreeBSD CDROM (fixit), because it uses a subset of the GENERIC kernel device drivers. If you can, try that before an actual emergency. So I probably don't need to pay attention to the instuctions talking about floppies. Not any more. A full FreeBSD install CD or DVD along with the fixit gives a lot more options. -Warren Block * Rapid City, South Dakota USA ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Best procedure for full backup of live system
Take a UFS2 snapshot and then backup the snapshot. A similar approach using ZFS snapshots would also work. See the handbook for more details. http://www.freebsd.org/doc/en/books/handbook/backup-strategies.html Nerius Landys wrote: My server is increasingly having important work stored on it, and I need to start taking backups of a lot of directories, especially /home, /opt, /etc, /usr/local/etc, and maybe others. The ideal backup (and what I've done in the past) is to take a full low-level dd image of the disk while the system is down (this is easy to do in a situation where you have dual boot). Or, since the output of dd would take up tons of space and would only be usable on an identical hard drive, use dump to take the backup while the machine is turned off (again easy to do on a dual boot). But now, I cannot bring down the machine. My plan is to do a tar gzip of / on the fly, and pipe that to ssh (remote machine). However, the system is live, and files will be in the progress of changing. My question is, what is the recommended procedure of taking a full backup on a live system? Ideally, if my hard drive were to crash, I would like to have such a backup so as to make it possible to copy over the entire backup to a new identical harddrive without doing any reinstall or configuration. Should I use tar/gzip? dump? What exact command should I use? I guess I'll back up all of / including system files, because there is not too much data. I will be piping the output to ssh. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Best procedure for full backup of live system
On Wed, 14 Oct 2009, Nerius Landys wrote: My question is, what is the recommended procedure of taking a full backup on a live system? Ideally, if my hard drive were to crash, I would like to have such a backup so as to make it possible to copy over the entire backup to a new identical harddrive without doing any reinstall or configuration. Should I use tar/gzip? dump? What exact command should I use? dump(8) with the -L option will take a snapshot of a live filesystem and then back that up. -Warren Block * Rapid City, South Dakota USA ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Best procedure for full backup of live system
Nerius Landys wrote: My server is increasingly having important work stored on it, and I need to start taking backups of a lot of directories, especially /home, /opt, /etc, /usr/local/etc, and maybe others. The ideal backup (and what I've done in the past) is to take a full low-level dd image of the disk while the system is down (this is easy to do in a situation where you have dual boot). Or, since the output of dd would take up tons of space and would only be usable on an identical hard drive, use dump to take the backup while the machine is turned off (again easy to do on a dual boot). But now, I cannot bring down the machine. My plan is to do a tar gzip of / on the fly, and pipe that to ssh (remote machine). However, the system is live, and files will be in the progress of changing. My question is, what is the recommended procedure of taking a full backup on a live system? Ideally, if my hard drive were to crash, I would like to have such a backup so as to make it possible to copy over the entire backup to a new identical harddrive without doing any reinstall or configuration. Should I use tar/gzip? dump? What exact command should I use? I guess I'll back up all of / including system files, because there is not too much data. I will be piping the output to ssh. Always consider, document and thoroughly test-utilize anything that 'Warren Block' has to say about backup and archiving ( search the archives for his name ). *always* remember that _archive_ is not a _backup_. *always* remember that if you have neither an archive or a backup, you are a complete failure as an 'admin' in general. *never* be responsible for losing data...EVER. I like to use space on remote servers ( or oftentimes local disks ) as clone-able space. To manufacture a live clone, I love rsync(1). Without getting into the nitty-gritty ( ( proper db mgmt pausing etc ), Here is a qd example to get the blood flowing.. comment the first line...legal..blah etc ): # rsync -arcvv \ # be very verbose --exclude=/backup \ # ignore our backup location --exclude=/tmp \ # ignore sessions, etc --delete-after \ # delete changes since last rsync / \ # sync this /backup # ...to this... Steve ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
