Re: continuous backup solution for FreeBSD
On Fri, 10 Oct 2008 08:42:49 -0700 Jeremy Chadwick [EMAIL PROTECTED] wrote: On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: On Fri, 10 Oct 2008 07:41:11 -0700 Jeremy Chadwick [EMAIL PROTECTED] wrote: On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: Mike Meyer wrote: On Fri, 10 Oct 2008 02:34:28 +0300 [EMAIL PROTECTED] wrote: Quoting Oliver Fromme [EMAIL PROTECTED]: These features are readily available right now on FreeBSD. You don't have to code anything. Well with 2 downsides, Once you actually try and implement these solutions, you'll see that your downsides are largely figments of your imagination. So if it is my imagination, how can I actually convert UFS to ZFS easily? Everybody seems to say that this is easy and that is easy. It's not that easy. I really don't know why people are telling you it is. Maybe because it is? Of course, it *does* require a little prior planning, but anyone with more than a few months experience as a sysadmin should be able to deal with it without to much hassle. Converting some filesystems are easier than others; /home (if you create one) for example is generally easy: 1) ZFS fs is called foo/home, mounted as /mnt 2) fstat, ensure nothing is using /home -- if something is, shut it down or kill it 3) rsync or cpdup /home files to /mnt 4) umount /home 5) zfs set mountpoint=/home foo/home 6) Restart said processes or daemons See! It's like I said! EASY! You can do this with /var as well. Yup. Of course, if you've done it that way, you're not thinking ahead, because: Now try /usr. Hope you've got /rescue available, because once /usr/lib and /usr/libexec disappear, you're in trouble. Good luck doing this in multi-user, too. Oops. You F'ed up. If you'd done a little planning, you would have realized that / and /usr would be a bit of extra trouble, and planned accordingly. And finally, the root fs. Whoever says this is easy is kidding themselves; it's a pain. Um, no, it wasn't. Of course, I've been doing this long enough to have a system set up to make this kind of thing easy. My system disk is on a mirror, and I do system upgrades by breaking the mirror and upgrading one disk, making everything work, then putting the mirror back together. And moving to zfs on root is a lot like a system upgrade: 1) Break the mirror (mirrors actually, as I mirrored file systems). 2) Repartition the unused drive into /boot, swap data 3) Build zfs /boot according to the instructions on ZFSOnRoot wiki, just copying /boot and / at this point. 4) Boot the zfs disk in single user mode. 5) If 4 fails, boot back to the ufs disk so you're operational while you contemplate what went wrong, then repeat step 3. Otherwise, go on to step 6. 6) Create zfs file systems as appropriate (given that zfs file systems are cheap, and have lots of cool features that ufs file systems don't have, you probably want to create more than you had before, doing thing like putting SQL serves on their own file system with appropriate blocking, etc, but you'll want to have figured all this out before starting step 1). 7) Copy data from the ufs file systems to their new homes, not forgetting to take them out of /etc/fstab. 8) Reboot on the zfs disk. 9) Test until you're happy that everything is working properly, and be prepared to reboot on the ufs disk if something is broken. 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, add the data partition to the zfs pool so it's mirrored, and you should have already been using swap. This is 10 steps to your easy 6, but two of the extra steps are testing you didn't include, and 1 of the steps is a failure recovery step that shouldn't be necessary. So - one more step than your easy process. Of course, the part you seem to be (intentionally?) forgetting: most people are not using gmirror. There is no 2nd disk. They have one disk with a series of UFS2 filesystems, and they want to upgrade. That's how I read Evren's how do I do this? You say it's easy... comment, and I think his viewpoint is very reasonable. Granted, most people don't think about system upgrades when they build a system, so they wind up having to do extra work. In particular, Evren is talking about spending thousands of dollars on proprietary software, not to mention the cost of the server that all this data is going to flow to, for a backup solution. Compared to that, the cost of a few spare disks and the work to install them are trivial. Yeah, this isn't something you do on a whim. On the other hand, it's not something that any competent sysadmin would consider a pain. For a
Re: continuous backup solution for FreeBSD
On Sat, Oct 11, 2008 at 01:07:44PM +0200, Danny Braniss wrote: On Sat, Oct 11, 2008 at 12:35:16PM +0200, Danny Braniss wrote: On Fri, 10 Oct 2008 08:42:49 -0700 Jeremy Chadwick [EMAIL PROTECTED] wrote: On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: On Fri, 10 Oct 2008 07:41:11 -0700 Jeremy Chadwick [EMAIL PROTECTED] wrote: On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: Mike Meyer wrote: On Fri, 10 Oct 2008 02:34:28 +0300 [EMAIL PROTECTED] wrote: Quoting Oliver Fromme [EMAIL PROTECTED]: These features are readily available right now on FreeBSD. You don't have to code anything. Well with 2 downsides, Once you actually try and implement these solutions, you'll see that your downsides are largely figments of your imagination. So if it is my imagination, how can I actually convert UFS to ZFS easily? Everybody seems to say that this is easy and that is easy. It's not that easy. I really don't know why people are telling you it is. Maybe because it is? Of course, it *does* require a little prior planning, but anyone with more than a few months experience as a sysadmin should be able to deal with it without to much hassle. Converting some filesystems are easier than others; /home (if you create one) for example is generally easy: 1) ZFS fs is called foo/home, mounted as /mnt 2) fstat, ensure nothing is using /home -- if something is, shut it down or kill it 3) rsync or cpdup /home files to /mnt 4) umount /home 5) zfs set mountpoint=/home foo/home 6) Restart said processes or daemons See! It's like I said! EASY! You can do this with /var as well. Yup. Of course, if you've done it that way, you're not thinking ahead, because: Now try /usr. Hope you've got /rescue available, because once /usr/lib and /usr/libexec disappear, you're in trouble. Good luck doing this in multi-user, too. Oops. You F'ed up. If you'd done a little planning, you would have realized that / and /usr would be a bit of extra trouble, and planned accordingly. And finally, the root fs. Whoever says this is easy is kidding themselves; it's a pain. Um, no, it wasn't. Of course, I've been doing this long enough to have a system set up to make this kind of thing easy. My system disk is on a mirror, and I do system upgrades by breaking the mirror and upgrading one disk, making everything work, then putting the mirror back together. And moving to zfs on root is a lot like a system upgrade: 1) Break the mirror (mirrors actually, as I mirrored file systems). 2) Repartition the unused drive into /boot, swap data 3) Build zfs /boot according to the instructions on ZFSOnRoot wiki, just copying /boot and / at this point. 4) Boot the zfs disk in single user mode. 5) If 4 fails, boot back to the ufs disk so you're operational while you contemplate what went wrong, then repeat step 3. Otherwise, go on to step 6. 6) Create zfs file systems as appropriate (given that zfs file systems are cheap, and have lots of cool features that ufs file systems don't have, you probably want to create more than you had before, doing thing like putting SQL serves on their own file system with appropriate blocking, etc, but you'll want to have figured all this out before starting step 1). 7) Copy data from the ufs file systems to their new homes, not forgetting to take them out of /etc/fstab. 8) Reboot on the zfs disk. 9) Test until you're happy that everything is working properly, and be prepared to reboot on the ufs disk if something is broken. 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, add the data partition to the zfs pool so it's mirrored, and you should have already been using swap. This is 10 steps to your easy 6, but two of the extra steps are testing you didn't include, and 1 of the steps is a failure recovery step that shouldn't be necessary. So - one more step than your easy process. Of course, the part you seem to be (intentionally?) forgetting: most people are not using gmirror. There is no 2nd disk. They have one disk with a series of UFS2 filesystems, and they want to upgrade. That's how I read Evren's how do I do this? You say it's easy... comment, and I think his viewpoint is very reasonable. Granted,
Re: continuous backup solution for FreeBSD
On Sat, Oct 11, 2008 at 12:35:16PM +0200, Danny Braniss wrote: On Fri, 10 Oct 2008 08:42:49 -0700 Jeremy Chadwick [EMAIL PROTECTED] wrote: On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: On Fri, 10 Oct 2008 07:41:11 -0700 Jeremy Chadwick [EMAIL PROTECTED] wrote: On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: Mike Meyer wrote: On Fri, 10 Oct 2008 02:34:28 +0300 [EMAIL PROTECTED] wrote: Quoting Oliver Fromme [EMAIL PROTECTED]: These features are readily available right now on FreeBSD. You don't have to code anything. Well with 2 downsides, Once you actually try and implement these solutions, you'll see that your downsides are largely figments of your imagination. So if it is my imagination, how can I actually convert UFS to ZFS easily? Everybody seems to say that this is easy and that is easy. It's not that easy. I really don't know why people are telling you it is. Maybe because it is? Of course, it *does* require a little prior planning, but anyone with more than a few months experience as a sysadmin should be able to deal with it without to much hassle. Converting some filesystems are easier than others; /home (if you create one) for example is generally easy: 1) ZFS fs is called foo/home, mounted as /mnt 2) fstat, ensure nothing is using /home -- if something is, shut it down or kill it 3) rsync or cpdup /home files to /mnt 4) umount /home 5) zfs set mountpoint=/home foo/home 6) Restart said processes or daemons See! It's like I said! EASY! You can do this with /var as well. Yup. Of course, if you've done it that way, you're not thinking ahead, because: Now try /usr. Hope you've got /rescue available, because once /usr/lib and /usr/libexec disappear, you're in trouble. Good luck doing this in multi-user, too. Oops. You F'ed up. If you'd done a little planning, you would have realized that / and /usr would be a bit of extra trouble, and planned accordingly. And finally, the root fs. Whoever says this is easy is kidding themselves; it's a pain. Um, no, it wasn't. Of course, I've been doing this long enough to have a system set up to make this kind of thing easy. My system disk is on a mirror, and I do system upgrades by breaking the mirror and upgrading one disk, making everything work, then putting the mirror back together. And moving to zfs on root is a lot like a system upgrade: 1) Break the mirror (mirrors actually, as I mirrored file systems). 2) Repartition the unused drive into /boot, swap data 3) Build zfs /boot according to the instructions on ZFSOnRoot wiki, just copying /boot and / at this point. 4) Boot the zfs disk in single user mode. 5) If 4 fails, boot back to the ufs disk so you're operational while you contemplate what went wrong, then repeat step 3. Otherwise, go on to step 6. 6) Create zfs file systems as appropriate (given that zfs file systems are cheap, and have lots of cool features that ufs file systems don't have, you probably want to create more than you had before, doing thing like putting SQL serves on their own file system with appropriate blocking, etc, but you'll want to have figured all this out before starting step 1). 7) Copy data from the ufs file systems to their new homes, not forgetting to take them out of /etc/fstab. 8) Reboot on the zfs disk. 9) Test until you're happy that everything is working properly, and be prepared to reboot on the ufs disk if something is broken. 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, add the data partition to the zfs pool so it's mirrored, and you should have already been using swap. This is 10 steps to your easy 6, but two of the extra steps are testing you didn't include, and 1 of the steps is a failure recovery step that shouldn't be necessary. So - one more step than your easy process. Of course, the part you seem to be (intentionally?) forgetting: most people are not using gmirror. There is no 2nd disk. They have one disk with a series of UFS2 filesystems, and they want to upgrade. That's how I read Evren's how do I do this? You say it's easy... comment, and I think his viewpoint is very reasonable. Granted, most people don't think about system upgrades when they build a system, so they wind up having to do extra work. In particular, Evren is talking about spending thousands of dollars on proprietary software, not to mention the cost of the
Re: continuous backup solution for FreeBSD
On Sat, Oct 11, 2008 at 12:35:16PM +0200, Danny Braniss wrote: On Fri, 10 Oct 2008 08:42:49 -0700 Jeremy Chadwick [EMAIL PROTECTED] wrote: On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: On Fri, 10 Oct 2008 07:41:11 -0700 Jeremy Chadwick [EMAIL PROTECTED] wrote: On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: Mike Meyer wrote: On Fri, 10 Oct 2008 02:34:28 +0300 [EMAIL PROTECTED] wrote: Quoting Oliver Fromme [EMAIL PROTECTED]: These features are readily available right now on FreeBSD. You don't have to code anything. Well with 2 downsides, Once you actually try and implement these solutions, you'll see that your downsides are largely figments of your imagination. So if it is my imagination, how can I actually convert UFS to ZFS easily? Everybody seems to say that this is easy and that is easy. It's not that easy. I really don't know why people are telling you it is. Maybe because it is? Of course, it *does* require a little prior planning, but anyone with more than a few months experience as a sysadmin should be able to deal with it without to much hassle. Converting some filesystems are easier than others; /home (if you create one) for example is generally easy: 1) ZFS fs is called foo/home, mounted as /mnt 2) fstat, ensure nothing is using /home -- if something is, shut it down or kill it 3) rsync or cpdup /home files to /mnt 4) umount /home 5) zfs set mountpoint=/home foo/home 6) Restart said processes or daemons See! It's like I said! EASY! You can do this with /var as well. Yup. Of course, if you've done it that way, you're not thinking ahead, because: Now try /usr. Hope you've got /rescue available, because once /usr/lib and /usr/libexec disappear, you're in trouble. Good luck doing this in multi-user, too. Oops. You F'ed up. If you'd done a little planning, you would have realized that / and /usr would be a bit of extra trouble, and planned accordingly. And finally, the root fs. Whoever says this is easy is kidding themselves; it's a pain. Um, no, it wasn't. Of course, I've been doing this long enough to have a system set up to make this kind of thing easy. My system disk is on a mirror, and I do system upgrades by breaking the mirror and upgrading one disk, making everything work, then putting the mirror back together. And moving to zfs on root is a lot like a system upgrade: 1) Break the mirror (mirrors actually, as I mirrored file systems). 2) Repartition the unused drive into /boot, swap data 3) Build zfs /boot according to the instructions on ZFSOnRoot wiki, just copying /boot and / at this point. 4) Boot the zfs disk in single user mode. 5) If 4 fails, boot back to the ufs disk so you're operational while you contemplate what went wrong, then repeat step 3. Otherwise, go on to step 6. 6) Create zfs file systems as appropriate (given that zfs file systems are cheap, and have lots of cool features that ufs file systems don't have, you probably want to create more than you had before, doing thing like putting SQL serves on their own file system with appropriate blocking, etc, but you'll want to have figured all this out before starting step 1). 7) Copy data from the ufs file systems to their new homes, not forgetting to take them out of /etc/fstab. 8) Reboot on the zfs disk. 9) Test until you're happy that everything is working properly, and be prepared to reboot on the ufs disk if something is broken. 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, add the data partition to the zfs pool so it's mirrored, and you should have already been using swap. This is 10 steps to your easy 6, but two of the extra steps are testing you didn't include, and 1 of the steps is a failure recovery step that shouldn't be necessary. So - one more step than your easy process. Of course, the part you seem to be (intentionally?) forgetting: most people are not using gmirror. There is no 2nd disk. They have one disk with a series of UFS2 filesystems, and they want to upgrade. That's how I read Evren's how do I do this? You say it's easy... comment, and I think his viewpoint is very reasonable. Granted, most people don't think about system upgrades when they build a system, so they wind up having to do extra work. In particular, Evren is talking about spending thousands of dollars on proprietary software, not to mention the cost of the server that all this data is going to flow to, for a backup solution. Compared to that, the cost of a few spare disks and
Re: continuous backup solution for FreeBSD
On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: Mike Meyer wrote: On Fri, 10 Oct 2008 02:34:28 +0300 [EMAIL PROTECTED] wrote: Quoting Oliver Fromme [EMAIL PROTECTED]: These features are readily available right now on FreeBSD. You don't have to code anything. Well with 2 downsides, Once you actually try and implement these solutions, you'll see that your downsides are largely figments of your imagination. So if it is my imagination, how can I actually convert UFS to ZFS easily? Everybody seems to say that this is easy and that is easy. It's not that easy. I really don't know why people are telling you it is. Converting some filesystems are easier than others; /home (if you create one) for example is generally easy: 1) ZFS fs is called foo/home, mounted as /mnt 2) fstat, ensure nothing is using /home -- if something is, shut it down or kill it 3) rsync or cpdup /home files to /mnt 4) umount /home 5) zfs set mountpoint=/home foo/home 6) Restart said processes or daemons See! It's like I said! EASY! You can do this with /var as well. Now try /usr. Hope you've got /rescue available, because once /usr/lib and /usr/libexec disappear, you're in trouble. Good luck doing this in multi-user, too. And finally, the root fs. Whoever says this is easy is kidding themselves; it's a pain. You get to make a new filesystem called /boot, and have all sorts of fun. It's really not a snap-fingers-voila thing, and I will gladly argue with anyone who thinks otherwise. Is it do-able though? Yes. -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Fri, 10 Oct 2008 07:41:11 -0700 Jeremy Chadwick [EMAIL PROTECTED] wrote: On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: Mike Meyer wrote: On Fri, 10 Oct 2008 02:34:28 +0300 [EMAIL PROTECTED] wrote: Quoting Oliver Fromme [EMAIL PROTECTED]: These features are readily available right now on FreeBSD. You don't have to code anything. Well with 2 downsides, Once you actually try and implement these solutions, you'll see that your downsides are largely figments of your imagination. So if it is my imagination, how can I actually convert UFS to ZFS easily? Everybody seems to say that this is easy and that is easy. It's not that easy. I really don't know why people are telling you it is. Maybe because it is? Of course, it *does* require a little prior planning, but anyone with more than a few months experience as a sysadmin should be able to deal with it without to much hassle. Converting some filesystems are easier than others; /home (if you create one) for example is generally easy: 1) ZFS fs is called foo/home, mounted as /mnt 2) fstat, ensure nothing is using /home -- if something is, shut it down or kill it 3) rsync or cpdup /home files to /mnt 4) umount /home 5) zfs set mountpoint=/home foo/home 6) Restart said processes or daemons See! It's like I said! EASY! You can do this with /var as well. Yup. Of course, if you've done it that way, you're not thinking ahead, because: Now try /usr. Hope you've got /rescue available, because once /usr/lib and /usr/libexec disappear, you're in trouble. Good luck doing this in multi-user, too. Oops. You F'ed up. If you'd done a little planning, you would have realized that / and /usr would be a bit of extra trouble, and planned accordingly. And finally, the root fs. Whoever says this is easy is kidding themselves; it's a pain. Um, no, it wasn't. Of course, I've been doing this long enough to have a system set up to make this kind of thing easy. My system disk is on a mirror, and I do system upgrades by breaking the mirror and upgrading one disk, making everything work, then putting the mirror back together. And moving to zfs on root is a lot like a system upgrade: 1) Break the mirror (mirrors actually, as I mirrored file systems). 2) Repartition the unused drive into /boot, swap data 3) Build zfs /boot according to the instructions on ZFSOnRoot wiki, just copying /boot and / at this point. 4) Boot the zfs disk in single user mode. 5) If 4 fails, boot back to the ufs disk so you're operational while you contemplate what went wrong, then repeat step 3. Otherwise, go on to step 6. 6) Create zfs file systems as appropriate (given that zfs file systems are cheap, and have lots of cool features that ufs file systems don't have, you probably want to create more than you had before, doing thing like putting SQL serves on their own file system with appropriate blocking, etc, but you'll want to have figured all this out before starting step 1). 7) Copy data from the ufs file systems to their new homes, not forgetting to take them out of /etc/fstab. 8) Reboot on the zfs disk. 9) Test until you're happy that everything is working properly, and be prepared to reboot on the ufs disk if something is broken. 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, add the data partition to the zfs pool so it's mirrored, and you should have already been using swap. This is 10 steps to your easy 6, but two of the extra steps are testing you didn't include, and 1 of the steps is a failure recovery step that shouldn't be necessary. So - one more step than your easy process. Yeah, this isn't something you do on a whim. On the other hand, it's not something that any competent sysadmin would consider a pain. For a good senior admin, it's a lot easier than doing an OS upgrade from source, which should be the next step up from trivial. mike -- Mike Meyer [EMAIL PROTECTED] http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O ascii ribbon campaign - stop html mail - www.asciiribbon.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: On Fri, 10 Oct 2008 07:41:11 -0700 Jeremy Chadwick [EMAIL PROTECTED] wrote: On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: Mike Meyer wrote: On Fri, 10 Oct 2008 02:34:28 +0300 [EMAIL PROTECTED] wrote: Quoting Oliver Fromme [EMAIL PROTECTED]: These features are readily available right now on FreeBSD. You don't have to code anything. Well with 2 downsides, Once you actually try and implement these solutions, you'll see that your downsides are largely figments of your imagination. So if it is my imagination, how can I actually convert UFS to ZFS easily? Everybody seems to say that this is easy and that is easy. It's not that easy. I really don't know why people are telling you it is. Maybe because it is? Of course, it *does* require a little prior planning, but anyone with more than a few months experience as a sysadmin should be able to deal with it without to much hassle. Converting some filesystems are easier than others; /home (if you create one) for example is generally easy: 1) ZFS fs is called foo/home, mounted as /mnt 2) fstat, ensure nothing is using /home -- if something is, shut it down or kill it 3) rsync or cpdup /home files to /mnt 4) umount /home 5) zfs set mountpoint=/home foo/home 6) Restart said processes or daemons See! It's like I said! EASY! You can do this with /var as well. Yup. Of course, if you've done it that way, you're not thinking ahead, because: Now try /usr. Hope you've got /rescue available, because once /usr/lib and /usr/libexec disappear, you're in trouble. Good luck doing this in multi-user, too. Oops. You F'ed up. If you'd done a little planning, you would have realized that / and /usr would be a bit of extra trouble, and planned accordingly. And finally, the root fs. Whoever says this is easy is kidding themselves; it's a pain. Um, no, it wasn't. Of course, I've been doing this long enough to have a system set up to make this kind of thing easy. My system disk is on a mirror, and I do system upgrades by breaking the mirror and upgrading one disk, making everything work, then putting the mirror back together. And moving to zfs on root is a lot like a system upgrade: 1) Break the mirror (mirrors actually, as I mirrored file systems). 2) Repartition the unused drive into /boot, swap data 3) Build zfs /boot according to the instructions on ZFSOnRoot wiki, just copying /boot and / at this point. 4) Boot the zfs disk in single user mode. 5) If 4 fails, boot back to the ufs disk so you're operational while you contemplate what went wrong, then repeat step 3. Otherwise, go on to step 6. 6) Create zfs file systems as appropriate (given that zfs file systems are cheap, and have lots of cool features that ufs file systems don't have, you probably want to create more than you had before, doing thing like putting SQL serves on their own file system with appropriate blocking, etc, but you'll want to have figured all this out before starting step 1). 7) Copy data from the ufs file systems to their new homes, not forgetting to take them out of /etc/fstab. 8) Reboot on the zfs disk. 9) Test until you're happy that everything is working properly, and be prepared to reboot on the ufs disk if something is broken. 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, add the data partition to the zfs pool so it's mirrored, and you should have already been using swap. This is 10 steps to your easy 6, but two of the extra steps are testing you didn't include, and 1 of the steps is a failure recovery step that shouldn't be necessary. So - one more step than your easy process. Of course, the part you seem to be (intentionally?) forgetting: most people are not using gmirror. There is no 2nd disk. They have one disk with a series of UFS2 filesystems, and they want to upgrade. That's how I read Evren's how do I do this? You say it's easy... comment, and I think his viewpoint is very reasonable. Yeah, this isn't something you do on a whim. On the other hand, it's not something that any competent sysadmin would consider a pain. For a good senior admin, it's a lot easier than doing an OS upgrade from source, which should be the next step up from trivial. I guess you have a very different definition of easy. :-) The above procedure, in no way shape or form, will be classified as easy by the user (or even junior sysadmin) community, I can assure you of that. I'll also throw this in the mix: the fact that we are *expecting* users to know how to do this is unreasonable. It's even *more* rude to expect that mid-level or senior SAs have to do it the hard way. Why? I'll explain: I'm an SA of 16+ years. I'm quite familiar with PBR/MBR, general disk partitioning, sectors vs. blocks, slices, filesystems, and
Re: continuous backup solution for FreeBSD
On Fri, 10 Oct 2008 18:51:26 +0300 Evren Yurtesen [EMAIL PROTECTED] wrote: The original question (which was lost) was if somebody who has technical knowledge and coding skills who can put r1soft into the right track so their software can support FreeBSD. Because r1soft is interested in supporting FreeBSD. I answered that, and you ignored it: If rlsoft is interested in hiring someone for this, they should sedn a message to freebsd-jobs. If they aren't interested in hiring someone, they need to release the technical information that would be required to do it, and then provide pointers to that information to the FreeBSD community (i.e. - by posting it here, rather than just posting trolls). mike -- Mike Meyer [EMAIL PROTECTED] http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O ascii ribbon campaign - stop html mail - www.asciiribbon.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
I just wanted to say thanks to all the replies to this thread. It has been insightful even though the suggestions I have received were not really answers to what I asked. I dont see any reason why we should continue to argue about if this can be done using ZFS or Hammer or any other filesystem. The fact is that it can be done eventually with something else if one has the will, time and money to put into it. One can even write his own filesystem code for it eh? The original question (which was lost) was if somebody who has technical knowledge and coding skills who can put r1soft into the right track so their software can support FreeBSD. Because r1soft is interested in supporting FreeBSD. Most of you probably know that FreeBSD support in products of commercial companies are scarce already. This is causing reductions in user base and popularity, although I am not saying that the more users are the better or I am not saying that popularity is everything. But you have to be able to combine one issue with another and see what the results might be in the long run. For example, wouldnt it be good for FreeBSD itself if Oracle supported FreeBSD? While I cant make Oracle to support FreeBSD, there is a company out there with a popular and quality product which is interested in supporting FreeBSD in their products. So why not take advantage of the situation? While I didnt think about some details when I sent the first post, some people thought that I would want somebody to code for r1soft for free so they can sell the software. No I do not and did not expect anybody to do anything for free. So, to summarize, if there is anybody who wants to help the 'actual' issue here, either by getting payed or just giving free hints and tips to r1soft then please contact me (I can forward your contact info) or with r1soft. Dag-Erling Smørgrav already told me that I can send his info and I have sent his contact info to r1soft for this issue. I hope we can now stop arguing about all the other things :) It wasnt my intention to start such arguments. Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Fri, 10 Oct 2008 08:42:49 -0700 Jeremy Chadwick [EMAIL PROTECTED] wrote: On Fri, Oct 10, 2008 at 11:29:52AM -0400, Mike Meyer wrote: On Fri, 10 Oct 2008 07:41:11 -0700 Jeremy Chadwick [EMAIL PROTECTED] wrote: On Fri, Oct 10, 2008 at 03:53:38PM +0300, Evren Yurtesen wrote: Mike Meyer wrote: On Fri, 10 Oct 2008 02:34:28 +0300 [EMAIL PROTECTED] wrote: Quoting Oliver Fromme [EMAIL PROTECTED]: These features are readily available right now on FreeBSD. You don't have to code anything. Well with 2 downsides, Once you actually try and implement these solutions, you'll see that your downsides are largely figments of your imagination. So if it is my imagination, how can I actually convert UFS to ZFS easily? Everybody seems to say that this is easy and that is easy. It's not that easy. I really don't know why people are telling you it is. Maybe because it is? Of course, it *does* require a little prior planning, but anyone with more than a few months experience as a sysadmin should be able to deal with it without to much hassle. Converting some filesystems are easier than others; /home (if you create one) for example is generally easy: 1) ZFS fs is called foo/home, mounted as /mnt 2) fstat, ensure nothing is using /home -- if something is, shut it down or kill it 3) rsync or cpdup /home files to /mnt 4) umount /home 5) zfs set mountpoint=/home foo/home 6) Restart said processes or daemons See! It's like I said! EASY! You can do this with /var as well. Yup. Of course, if you've done it that way, you're not thinking ahead, because: Now try /usr. Hope you've got /rescue available, because once /usr/lib and /usr/libexec disappear, you're in trouble. Good luck doing this in multi-user, too. Oops. You F'ed up. If you'd done a little planning, you would have realized that / and /usr would be a bit of extra trouble, and planned accordingly. And finally, the root fs. Whoever says this is easy is kidding themselves; it's a pain. Um, no, it wasn't. Of course, I've been doing this long enough to have a system set up to make this kind of thing easy. My system disk is on a mirror, and I do system upgrades by breaking the mirror and upgrading one disk, making everything work, then putting the mirror back together. And moving to zfs on root is a lot like a system upgrade: 1) Break the mirror (mirrors actually, as I mirrored file systems). 2) Repartition the unused drive into /boot, swap data 3) Build zfs /boot according to the instructions on ZFSOnRoot wiki, just copying /boot and / at this point. 4) Boot the zfs disk in single user mode. 5) If 4 fails, boot back to the ufs disk so you're operational while you contemplate what went wrong, then repeat step 3. Otherwise, go on to step 6. 6) Create zfs file systems as appropriate (given that zfs file systems are cheap, and have lots of cool features that ufs file systems don't have, you probably want to create more than you had before, doing thing like putting SQL serves on their own file system with appropriate blocking, etc, but you'll want to have figured all this out before starting step 1). 7) Copy data from the ufs file systems to their new homes, not forgetting to take them out of /etc/fstab. 8) Reboot on the zfs disk. 9) Test until you're happy that everything is working properly, and be prepared to reboot on the ufs disk if something is broken. 10) Reformat the ufs disk to match the zfs one. Gmirror /boot, add the data partition to the zfs pool so it's mirrored, and you should have already been using swap. This is 10 steps to your easy 6, but two of the extra steps are testing you didn't include, and 1 of the steps is a failure recovery step that shouldn't be necessary. So - one more step than your easy process. Of course, the part you seem to be (intentionally?) forgetting: most people are not using gmirror. There is no 2nd disk. They have one disk with a series of UFS2 filesystems, and they want to upgrade. That's how I read Evren's how do I do this? You say it's easy... comment, and I think his viewpoint is very reasonable. Granted, most people don't think about system upgrades when they build a system, so they wind up having to do extra work. In particular, Evren is talking about spending thousands of dollars on proprietary software, not to mention the cost of the server that all this data is going to flow to, for a backup solution. Compared to that, the cost of a few spare disks and the work to install them are trivial. Yeah, this isn't something you do on a whim. On the other hand, it's not something that any competent sysadmin would consider a pain. For a good senior admin, it's a lot easier than doing an OS upgrade from source, which should be the next step
Re: continuous backup solution for FreeBSD
Evren Yurtesen wrote: Shaun Amott wrote: On Tue, Oct 07, 2008 at 12:31:58AM +0300, Evren Yurtesen wrote: so FreeBSD could be supported also. As you can imagine, it is not only important that data can be restored when a box hardware failure etc. it is also important that data can be restored if deleted by accidents etc. While traditional backup programs provide this functionality, you cant really go back to 10 min or 1h ago, often they take daily backups and have to scan whole filesystem for changed files every time the backup is taken which stresses out the systems. This can (more or less) be achieved with snapshots: you can cheaply maintain old versions of the file system, and mount an old snapshot at any time. Hourly is about as fine-grained as you can expect though. The documentation says one cant do more than 20 snapshots. http://www.freebsd.org/doc/en/books/handbook/snapshots.html I wouldn't use UFS snapshots for this purpose. They have a few well-known disadvantages. However, ZFS snapshots should work very well for this. They're not limited to 20, and you can create them very quickly and with low overhead. You could create a new snapshot every 5 minutes if you want. Then you can use the zfs send command to produce a stream that contains the incremental differences between the snapshot and the previous snapshot, i.e. the stream represents the changes to the filesystem within the past five minutes (or whatever snapshot interval you choose). You can store that stream in a file, on backup medium, or send it with ssh to a different continent. Every once in a while you should make a full backup from a snapshot, of course. Basically it works like any other incremental backup mechanism, except that you can make the time interval between incremental backups very small (like five minutes in the above example), so you get a nearly continous backup solution. By the way, if you accidentally deleted something, you can of course simply copy it back from a snapshot. You can even revert the whole file system to a previous snapshot using the zfs rollback command. This is like going back in time. There is no need to touch your backups for that. These features are readily available right now on FreeBSD. You don't have to code anything. Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things. -- Doug Gwyn ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Zaphod Beeblebrox wrote: Oliver Fromme wrote: However, ZFS does exist on FreeBSD, and I think it wouldn't be impossible to add similar features to ZFS. Possibly even as a ZFS module? This might be something better addressed at the ZFS project level --- but the next question is: does FreeBSD support ZFS modules? I'm sorry I don't know. But also see my other reply regarding ZFS snapshots ans zfs send. Another possibility would be to extend gjournal by adding time stamps to journal transactions and a possibility to feed the journal to a pipe, socket or whatever. And of course a client-side implementation that does something useful with the journal stream. This might even be a good SoC project. Now this interests me. Firstly, I thought that gjournal might only be responsible for the meta-data (but I'm happy to be wrong on this point). Nope, gjournal handles _all_ data, i.e. meta data and file contents. Secondly, is it a) sufficient and b) efficient to attempt to time-travel UFS with the gjournal log? I almost don't dare to mention DragonFly BSD again, but they do have a UFS journaling implementation that does exactly that: http://leaf.dragonflybsd.org/cgi/web-man?command=mountctl http://leaf.dragonflybsd.org/cgi/web-man?command=jscan However, I think the implementation was abandoned because it was obsoleted by the development of the HAMMER file system. But the basic functionality is there and works. Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd IRIX is about as stable as a one-legged drunk with hypothermia in a four-hundred mile per hour wind, balancing on a banana peel on a greased cookie sheet -- when someone throws him an elephant with bad breath and a worse temper. -- Ralf Hildebrandt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Go here: http://forums.mysql.com/read.php?28,197644,197644 There are a ton of ways to maintain mysql backups without having to replay the entire log. Google some keywords. With regards to solutions based on filesystem snapshots, such as partial log replaying (snapshot + new-log, then replay from snapshot after crash), HAMMER and ZFS are your only real choices in the BSD world. UFS snapshots have all sorts of issues that make them unsuitable. Block level snapshots are unreliable. When interacting with a high level program for crash recovery purposes you really want to use a filesystem's native snapshot capability (e.g. HAMMER or ZFS. UFS's isn't good enough). You do not need to explicitly backup the filesystem to other media. That is, you of course still make such backups, but they would only be used in case of a catastrophic loss of the production filesystem, not for simple crash recovery which can be done from the same-media snapshots. -- In anycase, HAMMER has two native backup solutions. Both are easy to use. (1) Just use cpdup. This works nicely as the source filesystem, if HAMMER, can be snapshotted, and then used as a stable cpdup source. cpdup to the target and let HAMMER manage the history for you. No hardlink trick needed or anything like that. Use HAMMER's snapshots on the target to determine what history you want to retain. Advantages: Extremely convenient, target storage is reasonably efficient. Great if sources are mixed filesystem types. Disadvantages: Runs cpdup so you are scanning the directory hierarchy on both. Inode numbers not retained, so not suitable for fallback. (this sort of methodology can be used w/ ZFS too). (2) For HAMMER-to-HAMMER you can use HAMMER's native mirroring. Simply create a PFS (pseudo hammer filesystem) slave on the target, which takes 2 seconds, then use the 'hammer mirror-copy' or 'hammer mirror-stream' directives (which can take remote host specifications and will run ssh) to mirror from source to target. HAMMER's mirroring is incremental from the protocol right down to the media accesses. No pre-scan occurs is needed. Advantages: Provides bandwidth-controlled incremental streaming, near-realtime operation (30-60 second lag though finer-grained operation is possible). Extremely robust (can be ^C'd and restarted, or left offline for long periods of time, etc). Mirrors all fine-grained history on source and can be re-pruned on the target to the desired interval. Has little effect on production machines (no queues means backups can't feed-back and effect the performance of the production box). Mirrors inode numbers etc. Mirroring target can become a new master in a pinch (but can't be made a slave again, sorry). Should serve the same NFS fsid, etc. Disadvantages: Must mirror the entire PFS, only works between HAMMER filesystems. Near-real-time (30-60 second lag) is not real time, but it's probably close enough. (ZFS has a way to do something similar but I do not know what the various advantages or disadvantages of using the feature are). DragonFly also has real-time journaling at the VFS layer, which is not HAMMER-specific, but it requires queuing and isn't really suitable on a production environment for off-machine copying because the queue can fill up and block the filesystem. HAMMER's mirroring is far more robust. -Matt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Thu, Oct 9, 2008 at 3:40 PM, Matthew Dillon [EMAIL PROTECTED]wrote: (ZFS has a way to do something similar but I do not know what the various advantages or disadvantages of using the feature are). The only current way to do this on ZFS is to snapshot (very cheap) and stream the differences between the current snapshot and the previous snapshot to the remote host. The remote host can store the flat files or store the filesystem (that is: the streams can feed into new snapshots of the filesystem on the remote machine). Like Hammer, this gives history on both the local machine and the remote machine and is amazingly efficient. Unlike hammer, the process is not automated by the filesystem. You need a script that does zfs snaphot... followed by zfs send ... | ssh remote zfs receive ... --- such that each individual backup is a job rather than the connection approach you discribe for Hammer. Unlike Hammer, ZFS doesn't, by default, keep all history. I was speculating earlier that this might be possible to make as a ZFS module, though. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Quoting Oliver Fromme [EMAIL PROTECTED]: These features are readily available right now on FreeBSD. You don't have to code anything. Well with 2 downsides, The fact that I still would need to take full backups once in a while if I do this and Linux users do not have to because the CDP software on Linux does not need to do this. The software expires the old data automatically and you only need a full backup at first run only. The bigger problem is that I have to convert all my filesystems to ZFS. Can one convert UFS2 to ZFS easily even? I dont wanna spend a large part of the year doing such job while Linux users can just do this on 'any' filesystem they use. How am I suppose to compete with companies which use Linux otherwise if I am doing this sort of tasks all the time? Thanks for all the advices but my original question was if somebody can give inside information to a company(for example r1soft) which is writing CDP backup solutions so they could implement such solution on FreeBSD also. Do you know such person? I am not really looking for alternatives because there is none. You cant just expect commercial companies to convert to a new filesystem to add a feature which other OSes manage without going to such measures. Can you imagine the monetary cost if all FreeBSD users had to convert to ZFS (or another filesystem) to take near cdp level backups? This simply would make people think 'I wish I used Linux from the beginning'. Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Fri, 10 Oct 2008 02:34:28 +0300 [EMAIL PROTECTED] wrote: Quoting Oliver Fromme [EMAIL PROTECTED]: These features are readily available right now on FreeBSD. You don't have to code anything. Well with 2 downsides, Once you actually try and implement these solutions, you'll see that your downsides are largely figments of your imagination. The fact that I still would need to take full backups once in a while Every time you started backing up a new file system. This is a requirement of all backup solutions. After that, never again. The bigger problem is that I have to convert all my filesystems to ZFS. Can one convert UFS2 to ZFS easily even? I didn't have any trouble. And once you do it, you have advantages that the poor schmucks using Linux can only dream about: like self-healing file systems that are so cheap and easy to create that it makes sense to put each application or jail on it's own file system, one that's tuned for the application. The ability to set up raid and mirror devices without ever having to deal with LLVM (worth the cost of entry all by itself), not having to worry about allocating partitions, and - well, the list just goes on and on. Having converted to ZFS on my FreeBSD boxes, the only thing I feel for my clients still using Linux file systems is pity. this on 'any' filesystem they use I seriously doubt that it supports things like GMailFS. How am I suppose to compete with companies which use Linux otherwise if I am doing this sort of tasks all the time? Well, once you've done the conversion, by outproducing them while they waste time dealing with LLVM, partitions, and other such crap that ZFS frees you from. Thanks for all the advices but my original question was if somebody can give inside information to a company(for example r1soft) which is writing CDP backup solutions so they could implement such solution on FreeBSD also. Do you know such person? The only inside information here is held by the company (for example rlsoft) providing the CDP software. On the FreeBSD side, the source and documentation are all freely available to anyone who wants to look at it. But it doesn't matter how well you know FreeBSD, you aren't going to get anywhere unless you also you know what the software from that company needs in order to operate. If said company wanted to hire someone to either write this or to help get their people started working with FreeBSD, then the thing to do is send mail to [EMAIL PROTECTED] announcing the position. If they aren't interested in hiring someone, but hope to get it done for free, then they should set up a web page providing the technical details that someone who knows FreeBSD (or is willing to learn it) needs to do the job. If all you want is a CDP solution and you don't care where it comes from - well, you pretty much get the same two choices. It's an interesting enough problem that you might get someone to build it for free, but don't expect it to use proprietary software from some company that already provides such a thing for other systems. Nor - if you don't provide any incentive for meeting your requirements - should you expect it to actually meet them. mike -- Mike Meyer [EMAIL PROTECTED] http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O ascii ribbon campaign - stop html mail - www.asciiribbon.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
:The fact that I still would need to take full backups once in a while =20 :if I do this and Linux users do not have to because the CDP software =20 :on Linux does not need to do this. The software expires the old data =20 :automatically and you only need a full backup at first run only. You have to do this anyway. Nobody in their right mind trusts a single storage solution for all of their backups. CDP is a brute-force block-level continuous backup mechanism and while it works, to a point, it also has all the drawbacks that ANY block level backup system has... it is discontinuous from the high level filesystem structure and while you can pretty much guarantee that it is possible to recover from a disaster eventually, the filesystem you choose to run on top of that block device still matters a lot. On top of that being a client-server solution CDP is going to have some significant bottlenecks too. Even more telling is the fact that block level storage solutions tend to migrate corruption instead of detect it early. Big oopses can wind up stealthily winding their way through all your backups when you use a block-level solution. I stopped using block-level solutions almost 10 years ago... that's how little I trust them. Only a 'modern' linux filesystem (ext3 or the upcoming ext4, reiser, xfs, and few others), or something like ZFS or HAMMER, really have the ability to reliably recover from a point-in-time block level snapshot. Filesystems such as ZFS and HAMMER also give you insanely good snapshotting solutions that are far, FAR more flexible then what CDP gives you. You can upgrade between EXT filesystems without having to copy, but if you decided the best filesystem for you was one of the other many Linux filesystems, such as Reiser4 or XFS, and you were running EXT3, then you would have to copy. There are massive lists of pluses and minuses to each of the linux filesystem choices. Data expiration is a non-issue. You have to think about it either way. You have to test that the backup system actually works. You have to carefully control the backup policy and in particular not allow heavy disk I/O (such as a hacker DD'ing to your filesystem for 24 hours) to blow out your ability to recover the system. It requires time and effort no matter how well automated it is. :The bigger problem is that I have to convert all my filesystems to =20 :ZFS. Can one convert UFS2 to ZFS easily even? I dont wanna spend a =20 :large part of the year doing such job while Linux users can just do =20 :this on 'any' filesystem they use. How am I suppose to compete with =20 :companies which use Linux otherwise if I am doing this sort of tasks =20 :all the time? I converted our main developer machine from UFS to HAMMER in about 12 hours. 99% of that time simply waiting for the cpdup to finish copying a few hundred gigabytes from the old filesystem to the new. I think we're talking a few days here... time enough to learn how the filesystem works and play with it, make a few test copies (as you would with ANY new system that you did not have previous experienced with), and then do it for real. Linux users cannot just do this with a flip of a switch either. The new filesystem has to be constructed and the old data copied over. The old data set has to be retained, you cannot convert in-place, it's just too dangerous. It takes a certain amount of time to copy the data no matter what OS you are running, based on the amount of data you have. Some filesystem transitions such as going from ext2 to ext3 or ext3 to ext4 (if I remember right) are forwards compatible and do not require copying, but that sort of transition is NOT to a new filesystem, it's to a newer version of the same filesystem. With storage capacities increasing so quickly and mandating the replacement of whole disk subsystems (for running and electricity cost more then anything else, and more convenience, and less maintainance) it is a small convenience at best if you are going to copy the data anyway. Frankly, even if I were upgrading from ext2 to ext3 the last thing I would do is run it in-place. There's just too high a chance of software bugs creeping in. I would want a fresh ext3 and I would want my old ext2 data sitting on a shelf for a few days in case something were to go terribly wrong. I'd copy there too, just for safety. What is a few extra hours compared to blowing up the life-blood of your company? :I am not really looking for alternatives because there is none. You =20 :cant just expect commercial companies to convert to a new filesystem =20 :to add a feature which other OSes manage without going to such =20 :measures. Can you imagine the monetary cost if all FreeBSD users had =20 :to convert to ZFS (or another
Re: continuous backup solution for FreeBSD
Zaphod Beeblebrox [EMAIL PROTECTED] writes: Dag-Erling Smørgrav [EMAIL PROTECTED] writes: What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. Well... I hinted that a hammer port would be sufficient (although they need to finish their replication design) and I hinted that the hammer approach may be graftable to ZFS. Both reasonably large effort-wise (but probably within the scope of a single developer with sufficient time). No... you're so far off the mark it's not even funny, especially when it's been repeatedly pointed out to you. This is not a file system, it's a backup system. It's not designed to survive a disk crash or an accidental file deletion, it's designed to survive a direct missile strike on your colo center. To quote Wikipedia, CDP is a service that captures changes to data to a separate storage location - emphasis on separate. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Bakul Shah [EMAIL PROTECTED] writes: Dag-Erling Smørgrav [EMAIL PROTECTED] writes: What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. From their http://forum.r1soft.com/CDP.html page: [...] You completely missed the mark. I know what R1Soft's product is. What I want to know is what needs to be done to port it to FreeBSD. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Dag-Erling Smørgrav wrote: Zaphod Beeblebrox [EMAIL PROTECTED] writes: Dag-Erling Smørgrav [EMAIL PROTECTED] writes: What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. Well... I hinted that a hammer port would be sufficient (although they need to finish their replication design) and I hinted that the hammer approach may be graftable to ZFS. Both reasonably large effort-wise (but probably within the scope of a single developer with sufficient time). No... you're so far off the mark it's not even funny, especially when it's been repeatedly pointed out to you. This is not a file system, it's a backup system. It's not designed to survive a disk crash or an accidental file deletion, it's designed to survive a direct missile strike on your colo center. To quote Wikipedia, CDP is a service that captures changes to data to a separate storage location - emphasis on separate. FWIW, the HAMMER file system _does_ support replication to remote targets (thus separate). Unfortunately they call this feature mirroring, which is misleading at best. It's really rather a replication mechanism, much like the binlog of MySQL. It can be used for various purposes, including live mirroring, delayed mirroring, archiving, backup and point-in-time recovery. Well, of course, all of that doesn't help us at all because HAMMER doesn't exist on FreeBSD. However, ZFS does exist on FreeBSD, and I think it wouldn't be impossible to add similar features to ZFS. Another possibility would be to extend gjournal by adding time stamps to journal transactions and a possibility to feed the journal to a pipe, socket or whatever. And of course a client-side implementation that does something useful with the journal stream. This might even be a good SoC project. Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd File names are infinite in length, where infinity is set to 255 characters. -- Peter Collinson, The Unix File System ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
one answer... www.bakbone.com On Wed, Oct 8, 2008 at 6:20 PM, Oliver Fromme [EMAIL PROTECTED]wrote: Dag-Erling Smørgrav wrote: Zaphod Beeblebrox [EMAIL PROTECTED] writes: Dag-Erling Smørgrav [EMAIL PROTECTED] writes: What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. Well... I hinted that a hammer port would be sufficient (although they need to finish their replication design) and I hinted that the hammer approach may be graftable to ZFS. Both reasonably large effort-wise (but probably within the scope of a single developer with sufficient time). No... you're so far off the mark it's not even funny, especially when it's been repeatedly pointed out to you. This is not a file system, it's a backup system. It's not designed to survive a disk crash or an accidental file deletion, it's designed to survive a direct missile strike on your colo center. To quote Wikipedia, CDP is a service that captures changes to data to a separate storage location - emphasis on separate. FWIW, the HAMMER file system _does_ support replication to remote targets (thus separate). Unfortunately they call this feature mirroring, which is misleading at best. It's really rather a replication mechanism, much like the binlog of MySQL. It can be used for various purposes, including live mirroring, delayed mirroring, archiving, backup and point-in-time recovery. Well, of course, all of that doesn't help us at all because HAMMER doesn't exist on FreeBSD. However, ZFS does exist on FreeBSD, and I think it wouldn't be impossible to add similar features to ZFS. Another possibility would be to extend gjournal by adding time stamps to journal transactions and a possibility to feed the journal to a pipe, socket or whatever. And of course a client-side implementation that does something useful with the journal stream. This might even be a good SoC project. Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd File names are infinite in length, where infinity is set to 255 characters. -- Peter Collinson, The Unix File System ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Wed, Oct 8, 2008 at 4:14 AM, Dag-Erling Smørgrav [EMAIL PROTECTED] wrote: Zaphod Beeblebrox [EMAIL PROTECTED] writes: Dag-Erling Smørgrav [EMAIL PROTECTED] writes: What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. Well... I hinted that a hammer port would be sufficient (although they need to finish their replication design) and I hinted that the hammer approach may be graftable to ZFS. Both reasonably large effort-wise (but probably within the scope of a single developer with sufficient time). No... you're so far off the mark it's not even funny, especially when it's been repeatedly pointed out to you. This is not a file system, it's a backup system. It's not designed to survive a disk crash or an accidental file deletion, it's designed to survive a direct missile strike on your colo center. To quote Wikipedia, CDP is a service that captures changes to data to a separate storage location - emphasis on separate. Wow... thanks for the flame, but there's no reason that the device that is receiving the hammer replication couldn't be on the other side of the globe and there's no reason it couldn't be considered a backup. Part of the advantage of the structure that allows you to efficiently select for new changes allows you to do the same kind of *backup* as they claim. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Wed, Oct 8, 2008 at 7:20 AM, Oliver Fromme [EMAIL PROTECTED]wrote: Dag-Erling Smørgrav wrote: FWIW, the HAMMER file system _does_ support replication to remote targets (thus separate). Unfortunately they call this feature mirroring, which is misleading at best. It's really rather a replication mechanism, much like the binlog of MySQL. It can be used for various purposes, including live mirroring, delayed mirroring, archiving, backup and point-in-time recovery [thank-you for repeating that, BTW] However, ZFS does exist on FreeBSD, and I think it wouldn't be impossible to add similar features to ZFS. Possibly even as a ZFS module? This might be something better addressed at the ZFS project level --- but the next question is: does FreeBSD support ZFS modules? Another possibility would be to extend gjournal by adding time stamps to journal transactions and a possibility to feed the journal to a pipe, socket or whatever. And of course a client-side implementation that does something useful with the journal stream. This might even be a good SoC project. Now this interests me. Firstly, I thought that gjournal might only be responsible for the meta-data (but I'm happy to be wrong on this point). Secondly, is it a) sufficient and b) efficient to attempt to time-travel UFS with the gjournal log? ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Shaun Amott wrote: On Tue, Oct 07, 2008 at 12:31:58AM +0300, Evren Yurtesen wrote: so FreeBSD could be supported also. As you can imagine, it is not only important that data can be restored when a box hardware failure etc. it is also important that data can be restored if deleted by accidents etc. While traditional backup programs provide this functionality, you cant really go back to 10 min or 1h ago, often they take daily backups and have to scan whole filesystem for changed files every time the backup is taken which stresses out the systems. This can (more or less) be achieved with snapshots: you can cheaply maintain old versions of the file system, and mount an old snapshot at any time. Hourly is about as fine-grained as you can expect though. The documentation says one cant do more than 20 snapshots. http://www.freebsd.org/doc/en/books/handbook/snapshots.html Although 20 could be enough combined with a normal backup program. As far as I understand creating snapshots will consume disk space, and freeze the disk writes for a certain amount of time, every time (I read 5 seconds for 8GB system http://www.wave2.org/2007/10/08/mysql-snapshots-on-freebsd/ ) snapshot is created. The snapshots are stored in the local filesystem and it would require manually transferring the data to a remote machine. More importantly, as far as I understand, if the hard drive totally fails, there would be no way to restore a snapshot anymore unless we have a dump of the whole filesystem and first restore it and make sure everything is exactly at the right blocks in the drive. No? Although this probably could be worked out. In my opinion it requires a lot of work, Bbt thanks for the advice. Just that I would rather pay a small amount of fee and use Linux and use a continous backup software which works as easy as install and run. Which also provides utilities for easily recovering files or the whole filesystem or disk. Thanks again for pointing out snapshots. It is more or less suitable :) Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Outback Dingo wrote: one answer... www.bakbone.com http://www.bakbone.com Unfortunately if you check their compatibility matrix you can see that I have to use Linux to be able to do CDP :) http://www.bakbone.com/docs/NetVault_Backup_Supported_Platforms_October_2008.pdf or am I reading this wrong? Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Zaphod Beeblebrox wrote: On Wed, Oct 8, 2008 at 4:14 AM, Dag-Erling Smørgrav [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Zaphod Beeblebrox [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] writes: Dag-Erling Smørgrav [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] writes: What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. Well... I hinted that a hammer port would be sufficient (although they need to finish their replication design) and I hinted that the hammer approach may be graftable to ZFS. Both reasonably large effort-wise (but probably within the scope of a single developer with sufficient time). No... you're so far off the mark it's not even funny, especially when it's been repeatedly pointed out to you. This is not a file system, it's a backup system. It's not designed to survive a disk crash or an accidental file deletion, it's designed to survive a direct missile strike on your colo center. To quote Wikipedia, CDP is a service that captures changes to data to a separate storage location - emphasis on separate. Wow... thanks for the flame, but there's no reason that the device that is receiving the hammer replication couldn't be on the other side of the globe and there's no reason it couldn't be considered a backup. Part of the advantage of the structure that allows you to efficiently select for new changes allows you to do the same kind of *backup* as they claim. Wouldnt that device need to keep the whole filesystem? Like if you have 10 machines with 10x 1GB drives (lets say each used about 250gb), you will need 10TB disk space in the backup server? ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Wed, 8 Oct 2008, Evren Yurtesen wrote: Thanks again for pointing out snapshots. It is more or less suitable :) I'll just warn you that if you're planning to use snapshots for your own purposes, to first do an extensive stress test on a non-critical machine with backed up data. I've had a lot of problems with snapshots occasionally causing deadlocks which hang the machine. This was under 6.x but I had the same problem under many previous versions, so I don't necessarily expect that it's fixed. Also, while it's never happened to me, I've heard other people report data corruption. -- Nate Eldredge [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Wed, 08 Oct 2008 10:18:47 +0200 =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= [EMAIL PROTECTED] wrote: Bakul Shah [EMAIL PROTECTED] writes: Dag-Erling Sm=C3=B8rgrav [EMAIL PROTECTED] writes: What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. From their http://forum.r1soft.com/CDP.html page: [...] You completely missed the mark. I know what R1Soft's product is. What I want to know is what needs to be done to port it to FreeBSD. Sorry, my mindreader is broken. If this is what you really wanted to know, why not ask R1Soft? -hackers is not going to shed any light on the specifics of R1Soft's product. I replied to say that implementing a similar solution is not hard (I am sure you knew that too but I wasn't responding just to you). It may even be worth doing. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Wed, Oct 8, 2008 at 2:01 PM, Evren Yurtesen [EMAIL PROTECTED] wrote: Zaphod Beeblebrox wrote: Wow... thanks for the flame, but there's no reason that the device that is receiving the hammer replication couldn't be on the other side of the globe and there's no reason it couldn't be considered a backup. Part of the advantage of the structure that allows you to efficiently select for new changes allows you to do the same kind of *backup* as they claim. Wouldnt that device need to keep the whole filesystem? Like if you have 10 machines with 10x 1GB drives (lets say each used about 250gb), you will need 10TB disk space in the backup server? Urm... I think everything we've been discussing here backs up the whole filesystem (it would be near impossible for a block-oriented system to do elsewise). I suppose you could do something with the archive bit or dump bits with a filesystem based backup. But anyways... in a filesystem based replication system, you'd need enough space to store the data and the history of the data. The sum of the history of the data could even exceed the size of the sum of the input disks. It could also be much smaller. It really depends on how much you change. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Evren Yurtesen [EMAIL PROTECTED] writes: It would perhaps be too much to ask but can you drop a line to them and ask or is it ok if I give your email address to them and they can contact you? Feel free. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Evren Yurtesen [EMAIL PROTECTED] writes: Wouldnt that device need to keep the whole filesystem? Like if you have 10 machines with 10x 1GB drives (lets say each used about 250gb), you will need 10TB disk space in the backup server? Yes and no and yes... It stores *changes* to the file systems, so how much space it needs depends on how full your file systems are, how often and how much you write to them, etc. Also, at every recovery point, you can discard all but the last change since the previous recovery point for every changed block. FWIW, the exact same answer applies to pretty much any backup solution that supports incremental backups. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Bakul Shah [EMAIL PROTECTED] writes: Dag-Erling Smørgrav [EMAIL PROTECTED] writes: Bakul Shah [EMAIL PROTECTED] writes: Dag-Erling Smørgrav [EMAIL PROTECTED] writes: What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. From their http://forum.r1soft.com/CDP.html page: [...] You completely missed the mark. I know what R1Soft's product is. What I want to know is what needs to be done to port it to FreeBSD. Sorry, my mindreader is broken. If this is what you really wanted to know, why not ask R1Soft? -hackers is not going to shed any light on the specifics of R1Soft's product. I replied to say that implementing a similar solution is not hard (I am sure you knew that too but I wasn't responding just to you). It may even be worth doing. I didn't actually ask a question, and I don't mind that you don't have the answer. What I do mind is that you interpreted my statement of frustration as a question, and provided a completely irrelevant answer. You don't need to read minds to understand this, just English. If you take a step back and go through and read the entire thread again from the start, though, I think you will understand my frustration. Evren asked a question which everybody else is doing their best not to answer in as many words as possible; and when I try to answer, I find out that Evren doesn't really know what the question is. This is where - in an ideal world - somebody at R1Soft would jump in and start asking the right questions... DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Zaphod Beeblebrox [EMAIL PROTECTED] writes: Dag-Erling Smørgrav wrote: FWIW, the HAMMER file system _does_ support replication [...] No, actually, I didn't write that. You need to learn to quote properly. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Wed, 08 Oct 2008 22:19:48 +0200 =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= [EMAIL PROTECTED] wrote: Bakul Shah [EMAIL PROTECTED] writes: Dag-Erling Sm=C3=B8rgrav [EMAIL PROTECTED] writes: Bakul Shah [EMAIL PROTECTED] writes: Dag-Erling Sm=C3=B8rgrav [EMAIL PROTECTED] writes: What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. From their http://forum.r1soft.com/CDP.html page: [...] You completely missed the mark. I know what R1Soft's product is. What I want to know is what needs to be done to port it to FreeBSD. Sorry, my mindreader is broken. If this is what you really wanted to know, why not ask R1Soft? -hackers is not going to shed any light on the specifics of R1Soft's product. I replied to say that implementing a similar solution is not hard (I am sure you knew that too but I wasn't responding just to you). It may even be worth doing. I didn't actually ask a question, and I don't mind that you don't have the answer. What I do mind is that you interpreted my statement of frustration as a question, and provided a completely irrelevant answer. You don't need to read minds to understand this, just English. Interpreting an expression of frustration as a request for a solution is a common engineering trait:-) I can see you may not prefer my interpretation but I can't understand why you mind it. But so be it. I do not wish to annoy you. If you take a step back and go through and read the entire thread again from the start, though, I think you will understand my frustration. I understand your frustration but I chose to instead focus on the technical part. I too can get frustrated in similar situations but every time that happens I can trace it back to my own stress. I can't really control what others say so the way I deal with it is to ignore it or joke about it. I do try to clear misunderstandings but people don't always understand my point of view! As for feeling frustrated, I now view that as a warning signal. Evren asked a question which everybody else is doing their best not to answer in as many words as possible; and when I try to answer, I find out that Evren doesn't really know what the question is. Actually that was clear in his very first email. This is where - in an ideal world - somebody at R1Soft would jump in and start asking the right questions... Don't bet on it. My musings even makes me wonder if they do it right. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Sorry about that. Didn't mean to continue this discussion in -hackers but forgot to remove the cc list. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Quoting Zaphod Beeblebrox [EMAIL PROTECTED]: On Wed, Oct 8, 2008 at 2:01 PM, Evren Yurtesen [EMAIL PROTECTED] wrote: Zaphod Beeblebrox wrote: Wow... thanks for the flame, but there's no reason that the device that is receiving the hammer replication couldn't be on the other side of the globe and there's no reason it couldn't be considered a backup. Part of the advantage of the structure that allows you to efficiently select for new changes allows you to do the same kind of *backup* as they claim. Wouldnt that device need to keep the whole filesystem? Like if you have 10 machines with 10x 1GB drives (lets say each used about 250gb), you will need 10TB disk space in the backup server? Urm... I think everything we've been discussing here backs up the whole filesystem (it would be near impossible for a block-oriented system to do elsewise). I suppose you could do something with the archive bit or dump bits with a filesystem based backup. But anyways... in a filesystem based replication system, you'd need enough space to store the data and the history of the data. The sum of the history of the data could even exceed the size of the sum of the input disks. It could also be much smaller. It really depends on how much you change. The CDP backup solutions can manage that, even better you dont have to change to a totally new filesystem to use them. I really do not want to have a large argument about this but the only reason we cant do this on FreeBSD is because the companies who write CDP type backup solutions either do not know much about FreeBSD or they just cant find anybody who they can hire to do the job on FreeBSD. So that is why I posted this information to this list. If there is anybody who is capable of porting such software to FreeBSD then they can, 1- probably make money by doing the job, 2- if they have very little free time and not interested in making money then they can perhaps contact some of these companies (for example r1soft seems to be interested in supporting FreeBSD) and give them some hints and ideas on how this can be done on FreeBSD and where to find example codes and more information etc. Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Quoting Dag-Erling Smørgrav [EMAIL PROTECTED]: If you take a step back and go through and read the entire thread again from the start, though, I think you will understand my frustration. Evren asked a question which everybody else is doing their best not to answer in as many words as possible; and when I try to answer, I find out that Evren doesn't really know what the question is. This is where - in an ideal world - somebody at R1Soft would jump in and start asking the right questions... Actually, I was expecting that somebody would contact to R1Soft and offer them help for $$$. But I have given your contact information to them now. So they can forward their questions to you and probably you can let them know who knows what and who they can contact with more information or ask $$$ from them to do the job :) Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Hi, re Continuous Data Protection (an enterprise market niche, today), a recent paper that provides some background and has basic references might be: http://www.usenix.org/event/fast08/tech/verma.html SWEEPER: An Efficient Disaster Recovery Point Identification Mechanism, FAST '08, Akshat Verma, IBM India Research; Kaladhar Voruganti, Network Appliance; Ramani Routray, IBM Almaden Research; Rohit Jain, Yahoo India. -bruce ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Evren Yurtesen [EMAIL PROTECTED] writes: They actually do not think that it is an easy job to adapt their software to support FreeBSD even. See this post: http://forum.r1soft.com/showpost.php?p=4224postcount=3 All this shows is that they don't know anything about FreeBSD at all (plus they need a refresher course in OS design; Linux is also a monolithic kernel) What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
I think here might be a misunderstanding. I was talking about a reliable backup solution whereas you guys are all the time talking about mirroring and replication type solutions. Since you cant be thinking that mirroring and replication can replace backup, there must be a misunderstanding? Zaphod Beeblebrox wrote: From my reading, Hammer is much more than a filesystem, but then you probably havn't read about it yet. By my reading, Hammer hits all their feature points and does it better _because_ it's a filesystem. I glanced through these actually: http://www.dragonflybsd.org/hammer/ I didnt see anywhere that it will replace backup programs? It's relatively simple. Database replication solves the data backup problem (I don't store application data outside the database). Database replication for both MySQL and PostgreSQL is relatively straight forward. As for the configuration of code and servers --- that is taken care of with configuration management (it's really a bigger issue than just backing up a filesystem) and installing a new server to take a place in the cluster is a straight forward checkout from the CM system. For things I really care about staying up, add VRRP and an application design that is fault tolerant. Have you ever tried to restore data from MySQL replication logs? :) Even if you use binary logging, when you want to go back in time. You will need to first restore the whole database first from normal backups then replay the logs until the point that you wanna be. There is no simple way to go back in time. That is of course you have backups. If you dont have backups because you think replication is a backup solution, you would be screwed. Totally more complicated that clicking from the web to select data and time and table and restore! Also, you are thinking about a very small sized system. While replication might work if you are relatively small sized company (like 1-2 servers). If you have many independent servers with different databases inside you just cant use it. Even if you could replicate multiple boxes into one box, there would eventually be problems such as same named databases etc. and even then, you cant just easily restore the data if the user deletes all his data in his tables. Also that is not practical for users at all. For example I cant give an option to the user to restore his data by himself. While that is possible with most backup software easily. About VRRP etc. I already told that I am not talking about redundancy here. You are talking about totally different things. I need data protection. This actually works rather well if you do your research. Database replication is possible at all kinds of different link speeds and distances. Database replication also allows you to control your data better --- you know more about your data than a block replicator would. It means that your backup is already live and it means that, with the right scripts, invoking a backup on primary failure is simple. Database replication on some databases even allows you to preserve transactions --- which is important in some cases. And how do you propose that I restore a table in the database to of 1h before status? like you can do with a data backup solution? You are talking about a spare server solution not backup solution. Replication IS NOT backup. If you look at articles and information about database replication, almost all mention that it DOES NOT replace backups. Well... no. Backup software and strategies are just one available tool for risk mitigation. To know what tools you require, you must define your risks. Then with your list of risks you look at the cost of each tool and find the toolset that suits you. By the responses in this thread, it seems like the set of FreeBSD developers and the set of people who desire this solution are disjoint. Right, I just cant use the tool I require. There is no way to take near continuous backups of FreeBSD filesystems. Actually... as some obligatory positive content, the time travel features of Hammer should be straightforward to implement in ZFS... are ZFS modules supported on FreeBSD yet? It would seem to be a logical module. Those features work within the filesystem, you there is no mention of mirroring to a remote hard drive, if you could mirror to a remote hard drive, you couldnt easily mirror 10 machines into 1 remote hard drive, even if you could do that, it would require total disk space of all those 10 drives to exist in that 1 backup server. Even if you could overcome that, this would mean that all FreeBSD users wanting to take advantage of such system to convert their filesystems. Even if it was easy enough, there is no GUI tools to allow users exist in those systems to restore files by themselves easily. As you can see, none of the solutions you are suggesting is anywhere near simple solutions which can replace near continous backup
Re: continuous backup solution for FreeBSD
On Tue, Oct 07, 2008 at 12:31:58AM +0300, Evren Yurtesen wrote: so FreeBSD could be supported also. As you can imagine, it is not only important that data can be restored when a box hardware failure etc. it is also important that data can be restored if deleted by accidents etc. While traditional backup programs provide this functionality, you cant really go back to 10 min or 1h ago, often they take daily backups and have to scan whole filesystem for changed files every time the backup is taken which stresses out the systems. This can (more or less) be achieved with snapshots: you can cheaply maintain old versions of the file system, and mount an old snapshot at any time. Hourly is about as fine-grained as you can expect though. -- Shaun Amott // PGP: 0x6B387A9A A foolish consistency is the hobgoblin of little minds. - Ralph Waldo Emerson ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Tue, Oct 7, 2008 at 4:13 AM, Evren Yurtesen [EMAIL PROTECTED] wrote: Zaphod Beeblebrox wrote: From my reading, Hammer is much more than a filesystem, but then you probably havn't read about it yet. By my reading, Hammer hits all their feature points and does it better _because_ it's a filesystem. I glanced through these actually: http://www.dragonflybsd.org/hammer/ I didnt see anywhere that it will replace backup programs? You need to read the PDF on that page. On the first page of the PDF the 4th point is queue-less incremental mirroring If you read the PDF to determine what this phrase means, you'll find it describes the filesystem as a database indexed by a B+ tree of radix 64. It mentions that you can easily select all changes newer than a certain time. The problem with backup solutions that run on raw blocks is that you need meta-data to queue up the blocks that have changed faster than you can ship them out. Hammer solves this by allowing you to select changes based on when you shipped out the last batch of changes. The granularity of this will be dependent on how fast you can do this. It's relatively simple. Database replication solves the data backup problem (I don't store application data outside the database). Database replication for both MySQL and PostgreSQL is relatively straight forward. As for the configuration of code and servers --- that is taken care of with configuration management (it's really a bigger issue than just backing up a filesystem) and installing a new server to take a place in the cluster is a straight forward checkout from the CM system. For things I really care about staying up, add VRRP and an application design that is fault tolerant. Have you ever tried to restore data from MySQL replication logs? :) Even if you use binary logging, when you want to go back in time. You will need to first restore the whole database first from normal backups then replay the logs until the point that you wanna be. There is no simple way to go back in time. That is of course you have backups. If you dont have backups because you think replication is a backup solution, you would be screwed. Totally more complicated that clicking from the web to select data and time and table and restore! Largely why I use PostgreSQL. Also, you are thinking about a very small sized system. While replication might work if you are relatively small sized company (like 1-2 servers). If you have many independent servers with different databases inside you just cant use it. Even if you could replicate multiple boxes into one box, there would eventually be problems such as same named databases etc. and even then, you cant just easily restore the data if the user deletes all his data in his tables. Again, this is application design. I know of rather large organizations that use PostgreSQL replication to deliver very serious and onerous SLAs. Several of them have sponsored the Slovney (sp? russian for elephan --- the PostgreSQL multi-master replication thing). Also that is not practical for users at all. For example I cant give an option to the user to restore his data by himself. While that is possible with most backup software easily. Well... if we're talking about a random samba servers for windoze users, ZFS is the tech you want. Users can retrieve their own snapshots, etc. ZFS was designed with generalized fileserver duties in mind. And how do you propose that I restore a table in the database to of 1h before status? like you can do with a data backup solution? You are talking about a spare server solution not backup solution. Replication IS NOT backup. If you look at articles and information about database replication, almost all mention that it DOES NOT replace backups. I use differential backups to give old database restore functionality. I use replication to give me a live alternative database for failures (different risks, different solutions). Well... no. Backup software and strategies are just one available tool for risk mitigation. To know what tools you require, you must define your risks. Then with your list of risks you look at the cost of each tool and find the toolset that suits you. By the responses in this thread, it seems like the set of FreeBSD developers and the set of people who desire this solution are disjoint. Right, I just cant use the tool I require. There is no way to take near continuous backups of FreeBSD filesystems. You're free to pay anyone you'd like to implement this solution. I'd be happy to work for hire. I can provide a quote if you're serious about it. But if I was doing it for free, I'd want it to be fun and interesting. I'd be imlementing a server as well as a client. In fact, if I were doing this as my own open source project, I'd be looking at a better way of achieving it. FreeBSD is, after all, about doing things right. But --- as I said --- regardless of what I'd
Re: continuous backup solution for FreeBSD
I wanted to respond to DES' email separately --- because he's right. On Tue, Oct 7, 2008 at 5:56 AM, Dag-Erling Smørgrav [EMAIL PROTECTED] wrote: Evren Yurtesen [EMAIL PROTECTED] writes: They actually do not think that it is an easy job to adapt their software to support FreeBSD even. See this post: http://forum.r1soft.com/showpost.php?p=4224postcount=3 All this shows is that they don't know anything about FreeBSD at all (plus they need a refresher course in OS design; Linux is also a monolithic kernel) A very succinct way of making a similar poing :) What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. Well... I hinted that a hammer port would be sufficient (although they need to finish their replication design) and I hinted that the hammer approach may be graftable to ZFS. Both reasonably large effort-wise (but probably within the scope of a single developer with sufficient time). My primary concern about hammer is it's floating history stuff. It seems to me that legal compliance might have some things to say about it. It seems (from the document --- I havn't read the source) that the tunability of the real deletion of data are goals not absolutes. This is a concern. But as filesystems _are_ databases and as they grow database technology (like transactions and B+tree indexes), we should look to database systems for some of the solutions (ie: problems already solved). Replication replacing mirroring is just one of them. Doing this at the block level was suggested by someone earlier (BTW). Suggesting that a geom node could store a bit or two per bock (marking it dirty prehaps) and shipping off the blocks. That only solves the replication. You'd need something like a transaction ID per block stored on the backup server to enable time travel. Probably want a B+tree index there too. If you're not careful, you might find implementing the filesystem solution much easier. You could, however, implement this with hooks for gjournal --- such that the filesystems you backup are always sane. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Tue, 07 Oct 2008 11:56:09 +0200 =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= [EMAIL PROTECTED] wrote: Evren Yurtesen [EMAIL PROTECTED] writes: They actually do not think that it is an easy job to adapt their software to support FreeBSD even. See this post: http://forum.r1soft.com/showpost.php?p=3D4224postcount=3D3 All this shows is that they don't know anything about FreeBSD at all (plus they need a refresher course in OS design; Linux is also a monolithic kernel) What really annoys me with this thread is that nobody has provided any information at all that would allow someone to understand what needs to be done and estimate how hard it would be. From their http://forum.r1soft.com/CDP.html page: R1Soft's Continuous Data Protection solution is a ==near-Continuous Backups system capable of providing == hundreds of recovery points per day scheduled as little as 5 or 10 minutes apart. ... CDP Server works by reading your hard disk volumes at the sector level, bypassing the file system for the ultimate in performance and recovery. This disk sector synchronization is performed while the server is online and provides no interruption to other I/O requests even on a busy server. Clearly near-Continuous is *not* the same as continuous but never mind -- truthiness in business is so last century! But this could be the cause of some confusion. What they do is backups, not mirroring. A remote mirror would essentially require a continuous backup -- every disk write must be sent right away but in pure mirroring there is no access previous snapshots. In a true backup solution you can restore disk state to some number of previous backup points, regardless of whether you have *online* access to them. My guess is they have a driver that keeps track of disk writes. Something like set bit N of a bitmap when sector N is to be written. Then once every 10 minutes (or whatever snapshot interval you have selected) a client app scans the bitmap and sends these sectors to the backup server. If they did *just* this, there'd be consistency issues -- between the time a snapshot is taken and some sector N is actually backed up, there may be new writes to N by the OS. To deal with this, the new write must be delayed until N has been backed up. Another alternative is to slide forward the snapshot point. That is, if the snapshot was taken at time T1 and the backup finished by T2, and there were conflicting writes during [T1..T2), backup these writes as well and slide forward this snapshot time from T1 to T2. Repeat until there are no conflicting writes. This latter method won't block any writes from the OS. So my guess is they need an interface where they get notified for every disk write and optionally a way to delay a write. [To respond to an earlier point raised by others, I believe OS X Time Machine does a filesystem level backup and not at the disk level.] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Tue, Oct 7, 2008 at 11:50 AM, Shaun Amott [EMAIL PROTECTED] wrote: On Tue, Oct 07, 2008 at 12:31:58AM +0300, Evren Yurtesen wrote: so FreeBSD could be supported also. As you can imagine, it is not only important that data can be restored when a box hardware failure etc. it is also important that data can be restored if deleted by accidents etc. While traditional backup programs provide this functionality, you cant really go back to 10 min or 1h ago, often they take daily backups and have to scan whole filesystem for changed files every time the backup is taken which stresses out the systems. This can (more or less) be achieved with snapshots: you can cheaply maintain old versions of the file system, and mount an old snapshot at any time. Hourly is about as fine-grained as you can expect though. Unfortunately, as mentioned on another recent thread, FreeBSD UFS snapshots are quite fragile. The last time I tried this type-of-thing, it rather quickly resulted in filesystem corruption on moderately busy filesystems. Also, the cost of UFS snapshots (in time, while locking the filesystem) are unacceptable in practice. Now... ZFS seems to do this well enough. I use snapshots and send snapshots to my backup server --- it all seems to work OK. The idea that snapshots have infinite granularity and are automatic (introduced to me by the Hammer filesystem document) has merit, though --- and this is certainly core to the featurelist of the company that started this thread . ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
continuous backup solution for FreeBSD
Hello, As far as I can see, there is no continuous backup solution for FreeBSD at the moment. I talked with R1Soft and they seem to not be able to support FreeBSD and need help. Does anybody have free time and skills to give a hand? Please see: http://forum.r1soft.com/showpost.php?p=3414postcount=9 I know people who want to switch to using FreeBSD but cant do that because their backup solution (R1Soft) does not support FreeBSD :( Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Hi, On 6 Oct 2008, at 11:00, Evren Yurtesen wrote: As far as I can see, there is no continuous backup solution for FreeBSD at the moment. I talked with R1Soft and they seem to not be able to support FreeBSD and need help. Does anybody have free time and skills to give a hand? Please see: http://forum.r1soft.com/showpost.php?p=3414postcount=9 Should be possible to do this with a geom(4) class? I know people who want to switch to using FreeBSD but cant do that because their backup solution (R1Soft) does not support FreeBSD :( Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED] -- Bob Bishop [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Bob Bishop wrote: Does anybody have free time and skills to give a hand? Please see: http://forum.r1soft.com/showpost.php?p=3414postcount=9 Should be possible to do this with a geom(4) class? I am not saying it is impossible. They just need somebody to put them to right track I guess. I personally cant do that. It would be nice if somebody who has knowledge in this area contacts r1soft. At the very least r1soft seems to be willing to communicate on this issue. Continuous backups as well as bare-metal-restore seem to be a key feature for many hosters. FreeBSD is loosing users because of this issue. Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Mon, Oct 6, 2008 at 5:33 PM, Evren Yurtesen [EMAIL PROTECTED] wrote: Bob Bishop wrote: Does anybody have free time and skills to give a hand? Please see: http://forum.r1soft.com/showpost.php?p=3414postcount=9 Should be possible to do this with a geom(4) class? I am not saying it is impossible. They just need somebody to put them to right track I guess. I personally cant do that. It would be nice if somebody who has knowledge in this area contacts r1soft. At the very least r1soft seems to be willing to communicate on this issue. Continuous backups as well as bare-metal-restore seem to be a key feature for many hosters. FreeBSD is loosing users because of this issue. gmirror+ggate come to mind as a nifty solution ... Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED] -- ~/.signature: no such file or directory ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Mon, Oct 6, 2008 at 10:33 AM, Evren Yurtesen [EMAIL PROTECTED] wrote: [regarding r1soft.com, ...] I am not saying it is impossible. They just need somebody to put them to right track I guess. I personally cant do that. It would be nice if somebody who has knowledge in this area contacts r1soft. At the very least r1soft seems to be willing to communicate on this issue. Continuous backups as well as bare-metal-restore seem to be a key feature for many hosters. FreeBSD is loosing users because of this issue. Actually, having looked at the site, the hammer filesystem and it's replication strategy seem to be the most applicable technology (but then you wouldn't even need these guys --- you'd be doing it yourself). Like anything, though, live applications will require special treatment. Keeping a live filesystem replicated does in no way guarentee that your database (for instance) will be sane at any particular moment. It sounds like these guys have made allowances for MySQL (they specifically mention it), but this won't help the PostgreSQL users, etc. I've spent a lot of time thinking about redundancy and I've come to one inescapable conclusion: That the further up the stack you design for redundancy, the cheaper and easier it becomes. Most databases have replication strategies of one type or another that don't require exotic hosting solutions to work. The most fundamental example I can think of to show this principle, however, is the fact that if the HTTP standard required web browsers to try all A records (instead of randomly choosing one), web site redundancy would be amazingly simple to achieve. Consider that most other protocols right down to telnet do this, but web browsers don't. As a complete aside, if you have both and A records for your website, you have a form of poor-man's redundancy available to you --- with the caveat that it only works for people with both IPv6 and IPv4 connectivity. Browsers will try followed by A if the former doesn't respond. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On 12/23/-58 20:59, Evren Yurtesen wrote: Hello, As far as I can see, there is no continuous backup solution for FreeBSD at the moment. I talked with R1Soft and they seem to not be able to support FreeBSD and need help. Does anybody have free time and skills to give a hand? Please see: http://forum.r1soft.com/showpost.php?p=3414postcount=9 Quoting the thread: and they are usually very busy... and they like to work for hire. Wrong term: Even open source folks need something for living. That's too cheap and I don't suggest supporting a company like this, as they're expecting somebody else to do their job and make profit of it. They're free to use a system like FreeBSD for their products, they're free to modify and redistribute it but they need to do their job and don't expect anybody else to code for them for free. Next quote of thread: real world competency writing block device drivers for FreeBSD Let me check that out... %ls -l /dev | grep ^b % Well, I guess, nobody is able to do so and it's noted in the handbook or in GNN's The Design and Implementation of the FreeBSD operating system. A good read. I know people who want to switch to using FreeBSD but cant do that because their backup solution (R1Soft) does not support FreeBSD :( What's wrong with solutions like Bacula? It's working, cross platform, well supported. And now, go ahead and flame me for not doing work for free for other peoples profit but I'm pretty sure there won't be much guys wanting to do that. Volker ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Mon, 06 Oct 2008 18:09:06 +0300 Vlad GALU [EMAIL PROTECTED] wrote: On Mon, Oct 6, 2008 at 5:33 PM, Evren Yurtesen [EMAIL PROTECTED] wrote: Bob Bishop wrote: Does anybody have free time and skills to give a hand? Please see: http://forum.r1soft.com/showpost.php?p=3414postcount=9 Should be possible to do this with a geom(4) class? I am not saying it is impossible. They just need somebody to put them to right track I guess. I personally cant do that. It would be nice if somebod y who has knowledge in this area contacts r1soft. At the very least r1soft seems to be willing to communicate on this issue. Continuous backups as well as bare-metal-restore seem to be a key feature for many hosters. FreeBSD is loosing users because of this issue. gmirror+ggate come to mind as a nifty solution ... My guess is these guys do something simpler like keep keep track of changed blocks since the last backup and periodically dump those blocks to a server. This is good enough for backups (but not mirroring) and it has low memory overhead (1 or 2 bits per block), lower network overhead than remote mirroring (you send a block at most once every sync interval), and a tiny loss of performance (over no backups). May be someone ought to do a garchive device! ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Vlad GALU wrote: On Mon, Oct 6, 2008 at 5:33 PM, Evren Yurtesen [EMAIL PROTECTED] wrote: Bob Bishop wrote: Does anybody have free time and skills to give a hand? Please see: http://forum.r1soft.com/showpost.php?p=3414postcount=9 Should be possible to do this with a geom(4) class? I am not saying it is impossible. They just need somebody to put them to right track I guess. I personally cant do that. It would be nice if somebody who has knowledge in this area contacts r1soft. At the very least r1soft seems to be willing to communicate on this issue. Continuous backups as well as bare-metal-restore seem to be a key feature for many hosters. FreeBSD is loosing users because of this issue. gmirror+ggate come to mind as a nifty solution ... That allows mirroring however not a very practical solution. You would need to mirror every drive in another machine and you cant restore the drives into a earlier time. I am talking about a real backup solution where the backup agent collecting the information about written sectors and sending to backup server. You can then restore a file which existed in the box 1 hour ago if you need to. For example think about a situation where a server is processing important data which shouldnt get lost. A software failure wipes out the hard drive. You would loose all the data in the mirror as well. Also the setup of such mirroring system would be rather complicated. In addition to that, the mirroring does not support for example restore of mysql databases in table level. Think about a customer who wiped out his database accidentally. All the data would be gone in the mirror as well. With near continuous backup you can restore the data to just moments before the deletion process. Traditional backup systems at best daily backups, even if you could restore the data, the data could be up to 1 day old. More on that subject, r1soft supports multiple hosting control panel softwares. For example H-Sphere ( http://www.parallels.com/hsphere/ ) etc. through plugins which allow hosting customers to restore their own data easily. Something impossible with gmirror+ggate combination (since it does not actually backup the data and only mirror it) and not even practical if it was possible, if you have thousands of users. Actually, I am not saying that anybody should be doing about this and neither am I an r1soft advocate. I am just pointing out that there is a company out there which can provide a valuable software tool and they need somebody to put them into right direction only. If you know some FreeBSD developers who know the disk subsystem and can help R1Soft then you can perhaps forward this information to them. This not a feature or help request and I just am mentioning this. However you can imagine that a company which is giving online services on a serious scale probably would be interested in a CDP solution. It is currently impossible with FreeBSD. This makes using FreeBSD servers on critical applications a little bit insecure data protectionwise thus often people prefer Linux which is supported by such continuous data protection type backup solutions. (not that I am advocating FreeBSD or Linux here but this is the situation). Thanks, Evrne ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Zaphod Beeblebrox wrote: On Mon, Oct 6, 2008 at 10:33 AM, Evren Yurtesen [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: [regarding r1soft.com http://r1soft.com, ...] I am not saying it is impossible. They just need somebody to put them to right track I guess. I personally cant do that. It would be nice if somebody who has knowledge in this area contacts r1soft. At the very least r1soft seems to be willing to communicate on this issue. Continuous backups as well as bare-metal-restore seem to be a key feature for many hosters. FreeBSD is loosing users because of this issue. Actually, having looked at the site, the hammer filesystem and it's replication strategy seem to be the most applicable technology (but then you wouldn't even need these guys --- you'd be doing it yourself). Like anything, though, live applications will require special treatment. Keeping a live filesystem replicated does in no way guarentee that your database (for instance) will be sane at any particular moment. It sounds like these guys have made allowances for MySQL (they specifically mention it), but this won't help the PostgreSQL users, etc. I think you didnt get the point here. Replication or mirroring != backup. You cant return back to how things were 1 hour ago. Also they support postgresql as well (while its usage is way smaller than mysql) http://www.r1soft.com/CDP_db_postgreSQL.html In any case, the product guarantees that it can return your databases to any point in the time. Do you see what you are missing? :) I've spent a lot of time thinking about redundancy and I've come to one inescapable conclusion: That the further up the stack you design for redundancy, the cheaper and easier it becomes. Most databases have replication strategies of one type or another that don't require exotic hosting solutions to work. The idea/problem is not redundancy here, it is data protection. Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Bakul Shah writes: On Mon, 06 Oct 2008 18:09:06 +0300 Vlad GALU [EMAIL PROTECTED] wrote: On Mon, Oct 6, 2008 at 5:33 PM, Evren Yurtesen [EMAIL PROTECTED] wrote: Bob Bishop wrote: Does anybody have free time and skills to give a hand? Please see: http://forum.r1soft.com/showpost.php?p=3414postcount=9 Should be possible to do this with a geom(4) class? I am not saying it is impossible. They just need somebody to put them to right track I guess. I personally cant do that. It would be nice if somebod y who has knowledge in this area contacts r1soft. At the very least r1soft seems to be willing to communicate on this issue. Continuous backups as well as bare-metal-restore seem to be a key feature for many hosters. FreeBSD is loosing users because of this issue. gmirror+ggate come to mind as a nifty solution ... My guess is these guys do something simpler like keep keep track of changed blocks since the last backup and periodically dump those blocks to a server. This is good enough for backups (but not mirroring) and it has low memory overhead (1 or 2 bits per block), lower network overhead than remote mirroring (you send a block at most once every sync interval), and a tiny loss of performance (over no backups). May be someone ought to do a garchive device! There were a couple of threads about using kqueue or other FreeBSD tools to build something like Mac OS X's Time Machine. R1soft's software sounds very similar. The conclusion seemed to be that it'd be doable. Here's a pointer to the start of one of the threads. http://lists.freebsd.org/pipermail/freebsd-hackers/2008-June/024730.html g. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Sean Bruno wrote: Evren Yurtesen wrote: Hello, As far as I can see, there is no continuous backup solution for FreeBSD at the moment. I talked with R1Soft and they seem to not be able to support FreeBSD and need help. Does anybody have free time and skills to give a hand? Please see: http://forum.r1soft.com/showpost.php?p=3414postcount=9 I know people who want to switch to using FreeBSD but cant do that because their backup solution (R1Soft) does not support FreeBSD :( Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED] HmmmWell, my company (MiraLink) makes a BSD based appliance that mirrors block level changes between two units. It's commercial, not BSD licensed, but it's something. www.miralink.com sean Hello Sean, Thanks for the link. The miralink products seem to be doing data replication similar to gmirror+ggate? This wont protect against accidental deletion of data etc. The near continuous backup solutions do not mirror/replicate the data. Think about it as continuously taking backup. You can return back to 10 minutes or 1 hour before and restore old data. In either case, I wasnt trying to start a debate between backup solutions etc. here. I simply wanted to ask for help if somebody who has in depth knowledge about disk subsystems of FreeBSD and can give some tips to r1soft so FreeBSD could be supported also. As you can imagine, it is not only important that data can be restored when a box hardware failure etc. it is also important that data can be restored if deleted by accidents etc. While traditional backup programs provide this functionality, you cant really go back to 10 min or 1h ago, often they take daily backups and have to scan whole filesystem for changed files every time the backup is taken which stresses out the systems. Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Volker wrote: On 12/23/-58 20:59, Evren Yurtesen wrote: Hello, As far as I can see, there is no continuous backup solution for FreeBSD at the moment. I talked with R1Soft and they seem to not be able to support FreeBSD and need help. Does anybody have free time and skills to give a hand? Please see: http://forum.r1soft.com/showpost.php?p=3414postcount=9 Quoting the thread: and they are usually very busy... and they like to work for hire. Wrong term: Even open source folks need something for living. That's too cheap and I don't suggest supporting a company like this, as they're expecting somebody else to do their job and make profit of it. They're free to use a system like FreeBSD for their products, they're free to modify and redistribute it but they need to do their job and don't expect anybody else to code for them for free. I agree, perhaps whoever can help than can ask for money for the job done and I am sure they would pay reasonably since this is a commercial company. But as far as I can see people here do not even know the difference between near continous backup and mirroring. I just wanted to inform that there is such solutions nowadays available but FreeBSD users are not able to take advantage of them and the company who is making the product is interested in supporting FreeBSD but perhaps somebody who has experience can give some hints to them. They actually do not think that it is an easy job to adapt their software to support FreeBSD even. See this post: http://forum.r1soft.com/showpost.php?p=4224postcount=3 Next quote of thread: real world competency writing block device drivers for FreeBSD Let me check that out... %ls -l /dev | grep ^b % Well, I guess, nobody is able to do so and it's noted in the handbook or in GNN's The Design and Implementation of the FreeBSD operating system. A good read. You can go ahead and explain it to r1soft or any other software company which makes near continuous backup solutions. So maybe they can find this information and improve their product. So far at least this company thinks that this is impossible to do with FreeBSD as far as I can tell. I know people who want to switch to using FreeBSD but cant do that because their backup solution (R1Soft) does not support FreeBSD :( What's wrong with solutions like Bacula? It's working, cross platform, well supported. Do you know what is near continuous backup solution or ever heard of such technologies? I dont see how you can compare this with Bacula. They are not the same thing really. I can tell you one thing which is wrong with Bacula, it scans the whole filesystem everytime it takes a backup even incremental backups (while this disk io loading operation is unnecessary with CDP), and Bacula cant restore your data to what there was 10 minutes ago at any time of the day. And now, go ahead and flame me for not doing work for free for other peoples profit but I'm pretty sure there won't be much guys wanting to do that. There is nothing to flame or blame really. I can only blame my own stupidity. I posted this information here in hopes that somebody capable can give some hints to r1soft or to another near continuous backup solution so we could all benefit. But what do I get? All I get is information about solutions which mirror / replicate the data or suggestions of standard backup programs. I am not even surprised that FreeBSD is not supported. FreeBSD users do not even know the difference between a CDP solution or mirroring / replication or traditional backup... Go figure... If you dont know what is CDP then please read and learn (there are links to wikipedia articles in this address also) http://www.r1soft.com/CDP.html I never told that anybody should make the software ready for selling by another company for free, neither the post in the thread is asking for somebody to write a driver for free. The company obviously understands that such job requires an expert who works for money. They probably wouldnt mind if somebody wrote it for free to them (who wouldnt) but it doesnt say that they wouldnt pay. If you can do the job, please contact the company and give your price. If they tell that they want it done for free, then come and complain that they want it done for free. I am sorry if I was a little bit out of line here but a simple question became an unnecessary debate really. It was more or less a yes or no question :) Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Mon, Oct 6, 2008 at 3:38 PM, Evren Yurtesen [EMAIL PROTECTED] wrote: Zaphod Beeblebrox wrote: On Mon, Oct 6, 2008 at 10:33 AM, Evren Yurtesen [EMAIL PROTECTED]mailto: [EMAIL PROTECTED] wrote: [regarding r1soft.com http://r1soft.com, ...] I am not saying it is impossible. They just need somebody to put them to right track I guess. I personally cant do that. It would be nice if somebody who has knowledge in this area contacts r1soft. At the very least r1soft seems to be willing to communicate on this issue. Continuous backups as well as bare-metal-restore seem to be a key feature for many hosters. FreeBSD is loosing users because of this issue. Actually, having looked at the site, the hammer filesystem and it's replication strategy seem to be the most applicable technology (but then you wouldn't even need these guys --- you'd be doing it yourself). Like anything, though, live applications will require special treatment. Keeping a live filesystem replicated does in no way guarentee that your database (for instance) will be sane at any particular moment. It sounds like these guys have made allowances for MySQL (they specifically mention it), but this won't help the PostgreSQL users, etc. I think you didnt get the point here. Replication or mirroring != backup. You cant return back to how things were 1 hour ago. Actually, right back at you. You didn't fathom the meaning in my statement. While your post was vague, I read the company's website to determine the featureset they were claiming (although I missed their postgres support --- I only read the features page). NetBSD's hammer filesystem achieves replication at the filesystem layer (which will do infinitely better at this problem than a block-only driver) by maintaining a history of what's happened and being able to select (as in the database term) changes to the filesystem that have occurred since the last batch of blocks were shiped out to replication. This gives you both fine grained recovery (basically changes to files are kept until your rules define they should be freed) and replication and fine grained recovery on the other side of replication. In fact, Hammer delivers what they claim in a far more sophisticated way. (but it's NetBSD, not FreeBSD, unless someone decides to port it) Also they support postgresql as well (while its usage is way smaller than mysql) http://www.r1soft.com/CDP_db_postgreSQL.html In any case, the product guarantees that it can return your databases to any point in the time. Do you see what you are missing? :) Well... I am not missing it. I have that without making my filesystem jump through an enormous hoop. But designing my application correctly, I have that feature for far less effort. (that was the other point of my post) I've spent a lot of time thinking about redundancy and I've come to one inescapable conclusion: That the further up the stack you design for redundancy, the cheaper and easier it becomes. Most databases have replication strategies of one type or another that don't require exotic hosting solutions to work. The idea/problem is not redundancy here, it is data protection. Well... no... you don't need fine grained filesystem history for data integrity (unless you let loose a bunch of summer students armed with the ability to RM or your application is faulty (in which case, your filesystem won't protect you). As I said, This can all be achieved with other far simpler solutions. However, if your dev team isn't smart enough (or don't care for some reason), then you can take advantage of their product and pay their price. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Mon, 6 Oct 2008 14:24:32 -0700 George Hartzell [EMAIL PROTECTED] wrote: There were a couple of threads about using kqueue or other FreeBSD tools to build something like Mac OS X's Time Machine. R1soft's software sounds very similar. Time machine doesn't do continuous backups, it does them once an hour or so. People have built similar systems on top of rsync; I did it on top of zfs (turned out to be to fragile, though). You then just need a spiffy GUI for wondering through the backups. mike -- Mike Meyer [EMAIL PROTECTED] http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O ascii ribbon campaign - stop html mail - www.asciiribbon.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Zaphod Beeblebrox wrote: Actually, right back at you. You didn't fathom the meaning in my statement. While your post was vague, I read the company's website to I am sorry what was vague since I wrote 'continuous backup' in my post? The whole idea is such a basic idea that if you put this to google you can get wikipedia entry about it in first 3 results (I see in 2nd). Maybe you didnt know anything about it makes it vague to you. The message I sent was quite clear and plain. determine the featureset they were claiming (although I missed their postgres support --- I only read the features page). NetBSD's hammer filesystem achieves replication at the filesystem layer (which will do infinitely better at this problem than a block-only driver) by maintaining a history of what's happened and being able to select (as in the database term) changes to the filesystem that have occurred since the last batch of blocks were shiped out to replication. This gives you both fine grained recovery (basically changes to files are kept until your rules define they should be freed) and replication and fine grained recovery on the other side of replication. In fact, Hammer delivers what they claim in a far more sophisticated way. You might want to read this page too: http://www.r1soft.com/CDP.html (but it's NetBSD, not FreeBSD, unless someone decides to port it) While Hammer might be doing a similar job, it is a filesystem not a backup application and it wont replace backups. It doesnt just store the data in a backup server. While eventually it might become a backup solution, it will take years before that can happen. Even then, people will not just switch their current filesystems overnight. The CDP backup softwares are available today, it just needs a sort of driver to function in current system. Also they support postgresql as well (while its usage is way smaller than mysql) http://www.r1soft.com/CDP_db_postgreSQL.html In any case, the product guarantees that it can return your databases to any point in the time. Do you see what you are missing? :) Well... I am not missing it. I have that without making my filesystem jump through an enormous hoop. But designing my application correctly, I have that feature for far less effort. (that was the other point of my post) Can you just explain how can one currently do that in FreeBSD? Is it as easy as in Linux with CDP backup softwares? such as installing a program and done? I've spent a lot of time thinking about redundancy and I've come to one inescapable conclusion: That the further up the stack you design for redundancy, the cheaper and easier it becomes. Most databases have replication strategies of one type or another that don't require exotic hosting solutions to work. The idea/problem is not redundancy here, it is data protection. Well... no... you don't need fine grained filesystem history for data integrity (unless you let loose a bunch of summer students armed with the ability to RM or your application is faulty (in which case, your filesystem won't protect you). As I said, This can all be achieved with other far simpler solutions. However, if your dev team isn't smart enough (or don't care for some reason), then you can take advantage of their product and pay their price. There is no perfect system. This is exactly why people take backups. If what you said was applicable then there wouldnt be any need for backup software. People would just make sure that they dont loose their data. In addition to this, I have no control of if my customer will delete his/her data accidentally. I cant make a system which does not allow customers to delete data. I have given an example of a simple solution that Linux users can utilize (which obviously we also can utilize if we put our heads into it and give some directions as r1soft is willing to support FreeBSD). While you are first saying that these can be achieved with far simpler solutions, at the same time you are saying that the dev team must be smart enough. My solution is simple enough to write here. You install Linux on all machines and then CDP backup server on the backup server and CDP agent on the client machines, is yours simpler? Then explain how we can create similar sort of data protection? Thanks, Evren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
On Mon, Oct 6, 2008 at 6:42 PM, Evren Yurtesen [EMAIL PROTECTED] wrote: Zaphod Beeblebrox wrote: Actually, right back at you. You didn't fathom the meaning in my statement. While your post was vague, I read the company's website to While Hammer might be doing a similar job, it is a filesystem not a backup application and it wont replace backups. It doesnt just store the data in a backup server. While eventually it might become a backup solution, it will take years before that can happen. Even then, people will not just switch their current filesystems overnight. The CDP backup softwares are available today, it just needs a sort of driver to function in current system. From my reading, Hammer is much more than a filesystem, but then you probably havn't read about it yet. By my reading, Hammer hits all their feature points and does it better _because_ it's a filesystem. Well... I am not missing it. I have that without making my filesystem jump through an enormous hoop. But designing my application correctly, I have that feature for far less effort. (that was the other point of my post) Can you just explain how can one currently do that in FreeBSD? Is it as easy as in Linux with CDP backup softwares? such as installing a program and done? It's relatively simple. Database replication solves the data backup problem (I don't store application data outside the database). Database replication for both MySQL and PostgreSQL is relatively straight forward. As for the configuration of code and servers --- that is taken care of with configuration management (it's really a bigger issue than just backing up a filesystem) and installing a new server to take a place in the cluster is a straight forward checkout from the CM system. For things I really care about staying up, add VRRP and an application design that is fault tolerant. This actually works rather well if you do your research. Database replication is possible at all kinds of different link speeds and distances. Database replication also allows you to control your data better --- you know more about your data than a block replicator would. It means that your backup is already live and it means that, with the right scripts, invoking a backup on primary failure is simple. Database replication on some databases even allows you to preserve transactions --- which is important in some cases. There is no perfect system. This is exactly why people take backups. If what you said was applicable then there wouldnt be any need for backup software. People would just make sure that they dont loose their data. In addition to this, I have no control of if my customer will delete his/her data accidentally. I cant make a system which does not allow customers to delete data. I have given an example of a simple solution that Linux users can utilize (which obviously we also can utilize if we put our heads into it and give some directions as r1soft is willing to support FreeBSD). While you are first saying that these can be achieved with far simpler solutions, at the same time you are saying that the dev team must be smart enough. My solution is simple enough to write here. You install Linux on all machines and then CDP backup server on the backup server and CDP agent on the client machines, is yours simpler? Then explain how we can create similar sort of data protection? Well... no. Backup software and strategies are just one available tool for risk mitigation. To know what tools you require, you must define your risks. Then with your list of risks you look at the cost of each tool and find the toolset that suits you. By the responses in this thread, it seems like the set of FreeBSD developers and the set of people who desire this solution are disjoint. Actually... as some obligatory positive content, the time travel features of Hammer should be straightforward to implement in ZFS... are ZFS modules supported on FreeBSD yet? It would seem to be a logical module. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: continuous backup solution for FreeBSD
Mike Meyer writes: On Mon, 6 Oct 2008 14:24:32 -0700 George Hartzell [EMAIL PROTECTED] wrote: There were a couple of threads about using kqueue or other FreeBSD tools to build something like Mac OS X's Time Machine. R1soft's software sounds very similar. Time machine doesn't do continuous backups, it does them once an hour or so. People have built similar systems on top of rsync; I did it on top of zfs (turned out to be to fragile, though). You then just need a spiffy GUI for wondering through the backups. On the other hand Time Machine does take advantage of a kernel based mechanism that watches file activity and does its best to take advantage of that information to avoid scanning the filesystem when it does a backup. That's the context of the message thread that I pointed to (again, for completeness) http://lists.freebsd.org/pipermail/freebsd-hackers/2008-June/024730.html The thread seemed relevant given the context of backup systems that watch filesystem io. g. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
