Re: some thoughts about gmirror
--- Mike Wolman [EMAIL PROTECTED] wrote: It could also be used for asynchronous mirrors over slow links, if the log device was always written to first then the write latency for long distant links could be removed. Im not sure if it would be possible to achieve this using just a modified ggatec instead which has a local device used as a write cache. Sounds like rsync can already do that (I am not sure right now, if rsync can find updated areas within a large file, or if it just copies the while updated file even if it is a large one)... Furthermore the remote consumer of that gmirror couldnt be mounted RW, if it uses UFS, because UFS doesnt allow multiple RW mounts at the same time... -Arne Food fight? Enjoy some healthy debate in the Yahoo! Answers Food Drink QA. http://answers.yahoo.com/dir/?link=listsid=396545367 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: some thoughts about gmirror
R. B. Riddick wrote: Sounds like rsync can already do that (I am not sure right now, if rsync can find updated areas within a large file, or if it just copies the while updated file even if it is a large one)... rsync will find an updated area within a big file. The algorithm is to divide any such file into 100kB[*] chunks, calculate checksums of each of those chunks and only transfer the chunks where the checksum differs between source and destination. More detail here: http://samba.org/rsync/how-rsync-works.html http://rsync.samba.org/tech_report/tech_report.html Cheers, Matthew [*] For some value of 100kB. -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
Re: some thoughts about gmirror
Rsync is a great tool however if you try to rsync a filesystem with hundreds of thousand files in it the file list can use quite a large amount of bandwidth even if only a single file has changed - if you were keeping track of the blocks which had changed then you do not need to generate this list and simply send over the changed blocks. I was not thinking the remote side would mount the image unless the primary site was offline/unavailable. Mike. On Fri, 6 Apr 2007, R. B. Riddick wrote: --- Mike Wolman [EMAIL PROTECTED] wrote: It could also be used for asynchronous mirrors over slow links, if the log device was always written to first then the write latency for long distant links could be removed. Im not sure if it would be possible to achieve this using just a modified ggatec instead which has a local device used as a write cache. Sounds like rsync can already do that (I am not sure right now, if rsync can find updated areas within a large file, or if it just copies the while updated file even if it is a large one)... Furthermore the remote consumer of that gmirror couldnt be mounted RW, if it uses UFS, because UFS doesnt allow multiple RW mounts at the same time... -Arne Food fight? Enjoy some healthy debate in the Yahoo! Answers Food Drink QA. http://answers.yahoo.com/dir/?link=listsid=396545367 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: some thoughts about gmirror
Mike Wolman wrote: if you were keeping track of the blocks which had changed then you do not need to generate this list and simply send over the changed blocks. Unison keeps a list of files at each end and only exchanges block lists for files that have changed. I use it to sync 40GB (10K files) over a 1Mbps link and it's very fast. It also will do two-way sync. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: some thoughts about gmirror
if you were keeping track of the blocks which had changed then you do not need to generate this list and simply send over the changed blocks. Unison keeps a list of files at each end and only exchanges block lists for files that have changed. I use it to sync 40GB (10K files) over a 1Mbps link and it's very fast. It also will do two-way sync. Unison and rsync both work on the filesystem level and not with the blocks directly so would not be able to achieve the same result as the live network backup on netbsd - ie allowing a simple dd restore of a machine. As this would be filesystem independent and if you are running zfs or other snapshot capable filesystem i think rsync or unison would have a problem working with the snapshots. i do use rsync with close to about 1Tb of data and a lot hard links for - but if the remote file changes you have to store the entire copy of new file and not just the actual blocks which have changed. Mike. On Sat, 7 Apr 2007, Jim Rees wrote: Mike Wolman wrote: ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
some thoughts about gmirror
Hi, Currently I am using gmirror and ggated to run a live network mirror. Obviously this can cause problems if the server exporting the 'backup' device is offline then the mirror is broken - when the machines reconnect a full mirror sync takes place. This is fine over gbit crossover and if the size of the mirror is only a few 100Gb. Is it feasible that when the connection to one of the mirror devices breaks gmirror starts to log the changes to the mirror (obviously you would need to configure up this mirror device as a 'lazy' mirror member with a spare local device to write the changes to) - when the machines reconnect gmirror would only then have to sync the actual changes. This is sort of achieves a similar result to Live Network Backup on NetBSD (http://kerneltrap.org/node/5058). It could be used for laptop users mirroring their whole drive, allowing a fast sync when they are on their local lan and should the laptop get lost it would be possible to restore the whole machine with a simple dd. If they were using a usb key as the device to log the changes while they were disconnected from the network and they remember to unplug/plug this each time they use the laptop then it could even be possible to recover the data to the point they actually lost the machine. It could also be used for asynchronous mirrors over slow links, if the log device was always written to first then the write latency for long distant links could be removed. Im not sure if it would be possible to achieve this using just a modified ggatec instead which has a local device used as a write cache. Mike. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
