Re: OT: rsync on Mac OSX
On Jul 12, 2013, at 2:57 PM, kpn...@pobox.com wrote: > I thought MacOS X's rsync did handle resource forks if you gave it the > proper option. The resource fork is reported by rsync in the usual > convention of having "._" prefixed to the filename. My understanding was that the files named ._ were plain files that included the metadata that makes up the resource fork. The ._ file is not really the resource fork, but a workaround for filesystems that do not support resource forks. As such, they would be copied by rsync just fine. Now as to the Mac OS X rsync understanding resource forks, that I cannot speak to, but it should be easy to test. Copy a directory from an HFS+ volume to a non-Mac OS X volume (NFS for example) using rsync and see if it creates the ._ files to go with the data. -- Paul Kraus Deputy Technical Director, LoneStarCon 3 Sound Coordinator, Schenectady Light Opera Company ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: OT: rsync on Mac OSX
On Fri, Jul 12, 2013 at 11:25 AM, Chris Maness wrote: > Thank you for the detailed description of what resource forks are. One > more clue in this mystery is that appending .mov extension to it fixes the > problem. That makes some sense, since without the resource fork some MacOS software would have trouble identifying the type of the file. The extension provides that information in another way. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: OT: rsync on Mac OSX
On Fri, Jul 12, 2013 at 11:12 AM, Paul Kraus wrote: > Dropping the list … > > On Jul 12, 2013, at 1:49 PM, Chris Maness wrote: > > > Checksums are the same. All other files still work however the HUGE > > rendered Final Cut Pro output, so I guess it is something in .DS_Store. > > Last time I just gave up and recopied everything by a simple cut and > paste > > and that solved the problem. I made a small change on the project today, > > and I don't want to have to copy the WHOLE thing again just for a small > > delta. I already synced the directories, but the new rendered files are > > still un-openable in any application even though the checksums match. > > Really weird. However, the project will still open and work on FCP. > Just > > the 12Gb rendered movie files will not play on anything even FCP. If I > > delete .DS_Store will the system regenerate it with the appropriate file > > associations? > > The .DS_Store files are created by the Finder when you view a directory. > Are both source and destination on Mac HFS+ volumes ? If so, then you are > probably missing the resource forks. > > Back in the very old days of Mac OS (way before 10.x), Mac OS files had > two parts, the data part that contained the, well, data, and the resource > fork that contained the meta-data that Mac OS used to associate a file with > an application. HFS+ volumes on Mac OS X still include the resource forks, > but "foreign" filesystems (NFS, UFS, FAT, etc.) do not. The work around > that Apple came up with is to create .DS_Store and ._ files to store > this metadata on non HFS+ volumes. > > You could try using ditto instead of rsync. ditto is a BSD derived copy > utility similar to rysnc, but I know that the Mac OS X version understands > resource forks and copies them as necessary. ditto may not be able to just > copy changed blocks within a file, so you may still have to recopy the > entire file. > > But…. I am also a little puzzled because applications on Mac OS X do not > NEED the resource fork to open a file, just to know which application to > use (and what options to hand it) to open a given file. A complete video > file, even without resource forks, should be able to be opened if you > explicitly telly he application to File -> Open …. With the checksums > matching it is even odder. I expect that the large sizes (over 4 GB) are a > contributing factor. > > Good luck and let me know what you find. > > -- > Paul Kraus > Deputy Technical Director, LoneStarCon 3 > Sound Coordinator, Schenectady Light Opera Company > > Thank you for the detailed description of what resource forks are. One more clue in this mystery is that appending .mov extension to it fixes the problem. I have never ran into this before, and I have even used rsync to back up movie projects before. It is not a big deal, but I always try to take the time to understand why things behave the way they do. I also suspect it has something to do with file size since all of the smaller files do not have this issue. Thanks, Chris Maness ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: OT: rsync on Mac OSX
On 12 July 2013, at 10:49, Chris Maness wrote: > On Tue, Jul 2, 2013 at 3:11 PM, Chris Maness wrote: > >> >> >> >> >>> Since you are going to wait anyway, why don't you try peeking at some of >>> the file checksums while this is running? >>> >>> MacOS X comes with a shasum utility which implements SHA-256 checksums, >>> so you should be able to look at a few random samples of these files, >>> e.g. by running on the source disk: >>> >>>shasum -a 256 source_directory/file/path/to/some/file.ext >>> >>>shasum -a 256 copied_directory/file/path/to/some/file.ext >>> >>> If these are the same, then the applications look elsewhere, e.g. in the >>> 'hidden' .DS_Store stuff some MacOS directories contain. >>> >>> But if the checksums are different, well, then there's your problem. >>> >>> >> > Checksums are the same. All other files still work however the HUGE > rendered Final Cut Pro output, so I guess it is something in .DS_Store. > Last time I just gave up and recopied everything by a simple cut and paste > and that solved the problem. I made a small change on the project today, > and I don't want to have to copy the WHOLE thing again just for a small > delta. I already synced the directories, but the new rendered files are > still un-openable in any application even though the checksums match. > Really weird. However, the project will still open and work on FCP. Just > the 12Gb rendered movie files will not play on anything even FCP. If I > delete .DS_Store will the system regenerate it with the appropriate file > associations? > > I know this is a little off topic, but Mac OSX is based on BSD. You guys > are also the smartest around :D Rsync on the Mac only opens and copies the data forks. It does not copy the resource forks. There are still a few applications that use resource forks. Likewise the checksum apps work on the data forks only. There is a utility that is a modified rsync that does handle resource forks. I no longer remember what its name is. Its been a number of years since I last used it. I normally rsync from FreeBSD systems to Mac systems. I use Minis as off-site backups. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: OT: rsync on Mac OSX
On Tue, Jul 2, 2013 at 3:11 PM, Chris Maness wrote: > > > > >> Since you are going to wait anyway, why don't you try peeking at some of >> the file checksums while this is running? >> >> MacOS X comes with a shasum utility which implements SHA-256 checksums, >> so you should be able to look at a few random samples of these files, >> e.g. by running on the source disk: >> >> shasum -a 256 source_directory/file/path/to/some/file.ext >> >> shasum -a 256 copied_directory/file/path/to/some/file.ext >> >> If these are the same, then the applications look elsewhere, e.g. in the >> 'hidden' .DS_Store stuff some MacOS directories contain. >> >> But if the checksums are different, well, then there's your problem. >> >> > Checksums are the same. All other files still work however the HUGE rendered Final Cut Pro output, so I guess it is something in .DS_Store. Last time I just gave up and recopied everything by a simple cut and paste and that solved the problem. I made a small change on the project today, and I don't want to have to copy the WHOLE thing again just for a small delta. I already synced the directories, but the new rendered files are still un-openable in any application even though the checksums match. Really weird. However, the project will still open and work on FCP. Just the 12Gb rendered movie files will not play on anything even FCP. If I delete .DS_Store will the system regenerate it with the appropriate file associations? I know this is a little off topic, but Mac OSX is based on BSD. You guys are also the smartest around :D Thanks, Chris Maness ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: OT: rsync on Mac OSX
> Since you are going to wait anyway, why don't you try peeking at some of > the file checksums while this is running? > > MacOS X comes with a shasum utility which implements SHA-256 checksums, > so you should be able to look at a few random samples of these files, > e.g. by running on the source disk: > > shasum -a 256 source_directory/file/path/to/some/file.ext > > shasum -a 256 copied_directory/file/path/to/some/file.ext > > If these are the same, then the applications look elsewhere, e.g. in the > 'hidden' .DS_Store stuff some MacOS directories contain. > > But if the checksums are different, well, then there's your problem. > > That could be the issue. I did see some permission warnings with .DS_Store. However, to make space I had to dump all of it, so no files to do a post mortem on. I will have to check that out if things go awry this time. Thanks, Chris Maness ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: OT: rsync on Mac OSX
On Tue, 2 Jul 2013 14:48:03 -0700, Chris Maness wrote: >On Tue, Jul 2, 2013 at 2:30 PM, Giorgos Keramidas >wrote: >>On Tue, 2 Jul 2013 13:35:00 -0700, Chris Maness wrote: >>> I have been using rsync with Mac OSX with no issues until today. I >>> generally use it instead of the copy command because if the copy fails >>> on large files, I can pick up where I left off. I have backed up >>> entire Final Cut Pro projects this way with no issues. However, I >>> recently synced a drive to a folder in another drive, and the OS does >>> not recognize the final rendered files as quicktime files. The files >>> work fine in the parent drive. I have no idea what might be going on. >>> I used the flags: rsync -vaur like I always do. Any suggestions? >> >> This is a FreeBSD list, so any issues rsync may have with MacOS X are >> not very relevant to what FreeBSD is doing or would do. Having said >> that though, can you try without the -u option? Maybe modification >> times are newer on the target drive and rsync skips everything. >> >> You should probably also enable --stats and have a look at the final >> report of rsync, to see if it actually sync'ed any files, or skipped all >> of them because of mtime checks. > > Yep, the files copied, and I used "touch" to force them to recopy. > However, the files that were copied are not recognizable by their native > aps. Just big junk files. I have no clue what happened. I am just > copying everything by a simple cut and paste this time. However, this > directory is HUGE and I won't know until about 18 hours from now. Since you are going to wait anyway, why don't you try peeking at some of the file checksums while this is running? MacOS X comes with a shasum utility which implements SHA-256 checksums, so you should be able to look at a few random samples of these files, e.g. by running on the source disk: shasum -a 256 source_directory/file/path/to/some/file.ext shasum -a 256 copied_directory/file/path/to/some/file.ext If these are the same, then the applications look elsewhere, e.g. in the 'hidden' .DS_Store stuff some MacOS directories contain. But if the checksums are different, well, then there's your problem. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: OT: rsync on Mac OSX
On Tue, Jul 2, 2013 at 2:30 PM, Giorgos Keramidas wrote: > On Tue, 2 Jul 2013 13:35:00 -0700, Chris Maness > wrote: > > I have been using rsync with Mac OSX with no issues until today. I > > generally use it instead of the copy command because if the copy fails > > on large files, I can pick up where I left off. I have backed up > > entire Final Cut Pro projects this way with no issues. However, I > > recently synced a drive to a folder in another drive, and the OS does > > not recognize the final rendered files as quicktime files. The files > > work fine in the parent drive. I have no idea what might be going on. > > I used the flags: rsync -vaur like I always do. Any suggestions? > > This is a FreeBSD list, so any issues rsync may have with MacOS X are > not very relevant to what FreeBSD is doing or would do. Having said > that though, can you try without the -u option? Maybe modification > times are newer on the target drive and rsync skips everything. > > You should probably also enable --stats and have a look at the final > report of rsync, to see if it actually sync'ed any files, or skipped all > of them because of mtime checks. > > Yep, the files copied, and I used "touch" to force them to recopy. However, the files that were copied are not recognizable by their native aps. Just big junk files. I have no clue what happened. I am just copying everything by a simple cut and paste this time. However, this directory is HUGE and I won't know until about 18 hours from now. Thanks, Chris Maness ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: OT: rsync on Mac OSX
On Tue, 2 Jul 2013 13:35:00 -0700, Chris Maness wrote: > I have been using rsync with Mac OSX with no issues until today. I > generally use it instead of the copy command because if the copy fails > on large files, I can pick up where I left off. I have backed up > entire Final Cut Pro projects this way with no issues. However, I > recently synced a drive to a folder in another drive, and the OS does > not recognize the final rendered files as quicktime files. The files > work fine in the parent drive. I have no idea what might be going on. > I used the flags: rsync -vaur like I always do. Any suggestions? This is a FreeBSD list, so any issues rsync may have with MacOS X are not very relevant to what FreeBSD is doing or would do. Having said that though, can you try without the -u option? Maybe modification times are newer on the target drive and rsync skips everything. You should probably also enable --stats and have a look at the final report of rsync, to see if it actually sync'ed any files, or skipped all of them because of mtime checks. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
OT: rsync on Mac OSX
I have been using rsync with Mac OSX with no issues until today. I generally use it instead of the copy command because if the copy fails on large files, I can pick up where I left off. I have backed up entire Final Cut Pro projects this way with no issues. However, I recently synced a drive to a folder in another drive, and the OS does not recognize the final rendered files as quicktime files. The files work fine in the parent drive. I have no idea what might be going on. I used the flags: rsync -vaur like I always do. Any suggestions? Thanks, Chris Maness ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: RSync exclusion
Daniel O'Callaghan: It uses rsync, but manages a directory tree with hard links to unchanged files. It would solve your problem, but in a different way. Thanks for this - the issue is solved, but I will certainly have a look to that one as well. Jos ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: RSync exclusion
Must have had a temporary brain damage, sorry... thanks Jos Paul Macdonald: --exclude /files/photos ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: RSync exclusion
Hi, On Thu, 18 Apr 2013 13:30:45 +0200 Jos Chrispijn wrote: > I have a local folder called /files/ > In my daily backup event I create as per day a backkup folder that > contains this /files folder including all its sub folders > > --- cut --- > > rsync -avrz -e ssh /files/ backupr@x.x.x.x:/vol1/FreeBSD/$DATE/ > > In this /files folder, amongst others, I have another folder called > photos: /files/photos > > What I now would like to do is sync the /files folder with an > exclusion on the /files/photos folder I use the following option: --exclude '/data/Mail/*' /data/Mail is rooted in the directory I want to copy. Erich ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: RSync exclusion
In this /files folder, amongst others, I have another folder called photos: /files/photos What I now would like to do is sync the /files folder with an exclusion on the /files/photos folder Reason for that is that this /photos subfolder contains 12 gb on photos, which I don't want to have in a daily archive (takes too much disk space on a monthly basis). I will syn them on a daily basis to a fixed remote folder (where only the updates will be appended). Can someone tell me how I can do that on one command line? have you tried --exclude /files/photos Paul. -- - Paul Macdonald IFDNRG Ltd Web and video hosting - t: 0131 5548070 m: 07970339546 e: p...@ifdnrg.com w: http://www.ifdnrg.com - IFDNRG 40 Maritime Street Edinburgh EH6 6SA High Specification Dedicated Servers from £100.00pm ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: RSync exclusion
On 18/04/2013 9:30 PM, Jos Chrispijn wrote: rsync -avrz -e ssh /files/ backupr@x.x.x.x:/vol1/FreeBSD/$DATE/ Just a thought, but have you looked at rsnapshot? http://www.rsnapshot.org/ http://www.freshports.org/sysutils/rsnapshot/ It uses rsync, but manages a directory tree with hard links to unchanged files. It would solve your problem, but in a different way. regards, Danny ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
RSync exclusion
I have a local folder called /files/ In my daily backup event I create as per day a backkup folder that contains this /files folder including all its sub folders --- cut --- rsync -avrz -e ssh /files/ backupr@x.x.x.x:/vol1/FreeBSD/$DATE/ In this /files folder, amongst others, I have another folder called photos: /files/photos What I now would like to do is sync the /files folder with an exclusion on the /files/photos folder Reason for that is that this /photos subfolder contains 12 gb on photos, which I don't want to have in a daily archive (takes too much disk space on a monthly basis). I will syn them on a daily basis to a fixed remote folder (where only the updates will be appended). Can someone tell me how I can do that on one command line? thanks in advance, Jos Chrispijn ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Crash when trying to rsync to external NTFS-formatted HD
Waitman Gobble San Jose California USA On Mar 25, 2013 9:01 AM, "Lowell Gilbert" < freebsd-questions-lo...@be-well.ilk.org> wrote: > > Leslie Jensen writes: > > > I'm trying the following on my 9.1-RELEASE #0 system: > > > > Mounting the external HD > > ntfs-3g /dev/da0s1 /mnt/backup/ > > > > And doing > > rsync -av /home/les /mnt/backup/BSD_backup/ > > > > > > I get "Building incremental file list" for about one minute then my > > system freezes and reboots. > > > > I'm aware that NTFS might be the culprit. I have chosen it for > > convenience and the possibility to move my data to machines without > > FreeBSD. > > > > Manually copying of files one by one works. > > Yes, it definitely sounds like fuse is corrupting something in the > kernel. Could be tricky to debug, and forcing a kernel dump would be the > first step. > > The traditional most-portable way of moving files is to tar(1) onto a > raw device rather than having any filesystem at all. > ___ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " freebsd-questions-unsubscr...@freebsd.org" I have noticed this behavior with rsync and fuse mounted ntfs drives, hooking the drive up to an ms machine and doing a chkdisk type of operation may solve it for you, however i recommend using ufs formatted usb drives, it will ease headaches :) Waitman Gobble San Jose California ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Crash when trying to rsync to external NTFS-formatted HD
Leslie Jensen writes: > I'm trying the following on my 9.1-RELEASE #0 system: > > Mounting the external HD > ntfs-3g /dev/da0s1 /mnt/backup/ > > And doing > rsync -av /home/les /mnt/backup/BSD_backup/ > > > I get "Building incremental file list" for about one minute then my > system freezes and reboots. > > I'm aware that NTFS might be the culprit. I have chosen it for > convenience and the possibility to move my data to machines without > FreeBSD. > > Manually copying of files one by one works. Yes, it definitely sounds like fuse is corrupting something in the kernel. Could be tricky to debug, and forcing a kernel dump would be the first step. The traditional most-portable way of moving files is to tar(1) onto a raw device rather than having any filesystem at all. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Crash when trying to rsync to external NTFS-formatted HD
Hello list. I'm trying the following on my 9.1-RELEASE #0 system: Mounting the external HD ntfs-3g /dev/da0s1 /mnt/backup/ And doing rsync -av /home/les /mnt/backup/BSD_backup/ I get "Building incremental file list" for about one minute then my system freezes and reboots. I'm aware that NTFS might be the culprit. I have chosen it for convenience and the possibility to move my data to machines without FreeBSD. Manually copying of files one by one works. Thanks /Leslie ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: backups using rsync
On Tue, 05 Mar 2013 20:30:22 +0100, Matthias Petermann wrote: > Hello, > Zitat von Giorgos Keramidas : > >> If this is a UFS2 filesystem, it may be a good idea to snapshot the >> filesystem, and then rsync-backup the snapshot instead. > > Last time I tried UFS2 snapshots I found out two serious limitations. > The first is it doesn't work when UFS Journaling is used. The second is > that taking a snapshop on a large filesystem can cause parts of the > system to freeze for many minutes up to hours when accessing files > part of the snapshot, depending on the size of the filesystem. > That's why I could not use it on my server with > 1TB UFS2. > > Did this improve in the last year? (I guess my experience is from the > time around 9.0 release). Hi Matthias, Unfortunately I don't know if snapshots for such large filesystems are faster now. I've only used UFS2 snapshots in about 10x times smaller filesystems here. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: backups using rsync
On Mon, Mar 4, 2013 at 1:37 PM, CyberLeo Kitsana wrote: > You can use dump(8) to dump a SU-journaled filesystem; you just cannot > create a snapshot. This implies that dump(8) will be run against the > live and possibly changing filesystem, which can lead to issues with the > consistency of the contents of files thus dumped; but not necessarily > with the consistency of the dump itself. Any tool that backs up a live > filesystem, such as rsync or tar, will have these issues. > Note that this is mainly a problem for things like databases, where the contents of multiple files, or different portions of the same file, have to be in sync. For example, take your typical MySQL database table. You have the actual data, in a .MYD file, and the indexes, in a .MYI file. If your rsync backup hits while a table is being updated, it might get the .MYD file before an update, and the .MYI file after, leaving the table and index inconsistent. Or it might catch the .MYD file *partway* through an update, giving a file that's internally inconsistent. This is likely to give very unexpected results if you load the backup back into the database. Note that even if you take a filesystem snapshot, if you don't halt database updates while you take it, you can still end up with inconsistent files. Snapshots are mostly useful for limiting the downtime in these kinds of scenarios -- instead of taking the DB offline for the whole backup window, you just down it long enough to take the snapshot. In the absence of snapshots, the easiest way is to use whatever backup tools the database offers to make sure a consistent copy exists to be backed up. For example, before you run the backup, run mysqlhotcopy or mysqldump to write-lock the database, make consistent backup copies of all the files, then unlock it. That way, even if the backup of the active database is inconsistent, the copies that were backed up along with it are guaranteed to be consistent. Anything database-like can have this problem; another common example is a Subversion FSFS repository. Backing it up without running "svnadmin hotcopy" first is asking for corrupt commits when you do a restore. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: backups using rsync
Hello, Zitat von Giorgos Keramidas : If this is a UFS2 filesystem, it may be a good idea to snapshot the filesystem, and then rsync-backup the snapshot instead. Last time I tried UFS2 snapshots I found out two serious limitations. The first is it doesn't work when UFS Journaling is used. The second is that taking a snapshop on a large filesystem can cause parts of the system to freeze for many minutes up to hours when accessing files part of the snapshot, depending on the size of the filesystem. That's why I could not use it on my server with > 1TB UFS2. Did this improve in the last year? (I guess my experience is from the time around 9.0 release). Kind regards, Matthias -- Matthias Petermann ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: backups using rsync
On 2013-03-04 03:35, "Ronald F. Guilmette" wrote: > As a result of this past Black Friday weekend, I now enjoy a true > abundance of disk space, for the first time in my life. > > I wanna make a full backup, on a weekly basis, of my main system's > shiny new 1TB drive onto another 1TB drive that I also picked up cheap > back on Black Friday. > > I've been planning to set this up for some long time now, but I've > only gotten 'round to working on it now. > > Now, unfortunately, I have just been bitten by the evil... and > apparently widely known (except to me)... ``You can't use dump(8) to > dump a journaled filesystem with soft updates'' bug-a-boo. > > Sigh. The best laid plans of mice and men... > > I _had_ planned on using dump/restore and making backups from live mounted > filesystems while the system was running. But I really don't want to have > to take the system down to single-user mode every week for a few hours while > I'm making my disk-to-disk backup. So now I'm looking at doing the backups > using rsync. Yes, this should be possible... One thing that can bite you when using rsync to traverse & copy large filesystems is that the filesystem may still be changing beneath rsync *as it's doing* the copy. If this is a UFS2 filesystem, it may be a good idea to snapshot the filesystem, and then rsync-backup the snapshot instead. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: backups using rsync
On Mon, 04 Mar 2013 12:19:09 -0800, Ronald F. Guilmette wrote: > > In message <20130304125634.8450cfaf.free...@edvax.de>, > Polytropon wrote: > > >On Mon, 04 Mar 2013 03:35:30 -0800, Ronald F. Guilmette wrote: > >> Now, unfortunately, I have just been bitten by the evil... and apparently > >> widely known (except to me)... ``You can't use dump(8) to dump a journaled > >> filesystem with soft updates'' bug-a-boo. > > > >There are other tools you can use, for example tar or cpdup > >or rsync, as you've mentioned in the subject. > > tar I already knew about, but I think you will agree that it has lots of > limitations that make it entirely inappropriate for mirroring an entire > system. That's true. If your purpose is "backup of data files", tar is a good tool, especially for cross-platform use. But if you need to deal with "exceptional" things like extended permissions, ACL, sparse files and such, you will quickly see its limits. On the other hand, it can be used for multi-volume savesets, but this is not your intention. > This cpdup thing is entirely new to me. Thanks for mentioning it! I really > never heard of it before, but I just now installed it from ports, and I'm > perusing the man page. It's a little bit comparable to rsync and can also do things like "only add" (so you won't lose any files: if they are removed in source, they will be kept in backup). It also has limitations that rsync will not. > It looks very promising. Too bad it doesn't > properly handle sparse files, but oh well. That's just a very minor nit. > (Does it properly handle everything else that rsync claims to be able to > properly handle, e.g. ACLs, file attributes, etc., etc.?) That's something you should check with an "example dataset" you back up, restore, and compare. I've been using it for "normal files" successfully. > >The same problems that apply when dumping live systems can > >bite you using rsync, > > What problems are we talking about, in particular? The problems I'm refering to is the kind of _possible_ trouble you can get into when backing up files that keep changing. The ability to make a snapshot prior to starting the backup is a great help here (if you don't have the chance to unmount the partitions to backup). I can't imagine _how_ programs will react if they start reading a file, prepare sufficient space in some kind of TOC, then continue reading while the file grows... or if a file is being read which is removed during read... If you minimize the writing activity to the (still) _live_ data you're dealing with, that could be a benefit. > I am guessing that if I use rsync, then I *won't* encounter this rather > annoying issue/problem relating to UFS filesystems that have both soft > updates and journaling enabled, correct? > > >but support for this "on file system > >level" seems to be better in rsync than what dump does "on > >block level". > > What exactly did you mean by "this" ? As mentioned above: Unexpected and unpredictable results, strange kinds of inconsistency, may they appear during backup or later on restore. > >> If I use all of the following rsync options... -a,-H,-A, -X, and -S > >> when trying to make my backups, and if I do whatever additional fiddling > >> is necessary to insure that I separately copy over the MBR and boot loader > >> also to my backup drive, then is there any reason that, in the event of > >> a sudden meteor shower that takes out my primary disk drive while leaving > >> my backup drive intact, I can't just unplug my old primary drive, plug in > >> my (rsync-created) backup drive, reboot and be back in the sadddle again, > >> almost immediately, and with -zero- problems? > > > >You would have to make sure _many_ things are consistent > >on the backup disk. > > Well, this is what I am getting at. This is/was the whole point of my post > and my question. I want to know: What is that set of things, exactly? The backup disk (or failover disk, as I said) needs to be initialized properly prior to the first backup run: Make sure it's bootable. Depending on how you handle identification of the disk (by device name, by label or UFSID) and how you're going to boot from it (by selecting the failover disk in some post-BIOS/POST dialog or by swapping cables or bays), you should check it actually starts booting. > >Regarding terminology, that would make the disk a failover disk > > OK. Thank you. I will henceforth use that terminology. Just a suggestion from how you described you will be using the
Re: backups using rsync
On Mon, 4 Mar 2013, Ronald F. Guilmette wrote: So, um, I was reading about this last night, but I was sleepy and my eyes glazed over... Please remind me, what is the exact procedire for turning off the journaling? I boot to single user mode (from a live cd?) and then what? Is it tunefs with some special option? Just boot in single user mode so all the filesystems are unmounted or mounted readonly. Then use 'tunefs -j disable /dev/...'. It will also mention the name of the journal file, which can be deleted. Use the latest net/rsync port, and enable the FLAGS option. I use these options, copying each filesystem individually: -axHAXS --delete --fileflags --force-change Hummm... I guess that I have some non-current rsync installed. In the man page I have there is no mention of any "--force-change" option. What does it do? "affect user/system immutable files/dirs". Probably only included in the man page when the port is built with the FLAGS option set. An additional note: the script that runs my rsync backup also modifies the mirrored /etc/fstab to use the appropriate labels for the backup filesystems. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: backups using rsync
- Original Message - > On Mon, 04 Mar 2013 03:35:30 -0800, Ronald F. Guilmette wrote: > > Now, unfortunately, I have just been bitten by the evil... and > > apparently > > widely known (except to me)... ``You can't use dump(8) to dump a > > journaled > > filesystem with soft updates'' bug-a-boo. > > There are other tools you can use, for example tar or cpdup > or rsync, as you've mentioned in the subject. > > Or if you want to be ambitious you could install something like 'sysutils/backuppc' (where one of its methods is rsync, its what I use for all the systems I back up with it. - Windows, Linux, Mac OSX.) And, then could get more than just the weekly rsync to itthough it could probably be made to only do fulls every week. But, you could potentially then restore from an older full. I do system fulls of my other systems to it...can't do a baremetal restore, but it can get me back up and running faster. IE: I recently had harddrive failures in a couple of FreeBSD systems. I did a fresh install and at first I restored /home and /usr/local (and some other dirs, like /var/db/pkg & /var/db/ports)...and then other dirs and files as I found things missing. Had to rebuild a handful of ports after that and then things were good. The second system didn't go as well, because it had been silently corrupting things for a long time beforebut I still did the same kind of restore at first, but ended up rebuilding all the ports to get things good again. Not sure if losing the system disk, if I could recover from a local backuppc... but I have my old backuppc system, getting most of my current system (mainly omit the backuppc pool, think my backup storage requirements would grow exponentially if I didn'tmy main backuppc pool is currently 6300G out of 7300G zpool.) But, I've suffered bit rot on the old backuppc pool in the past..when it was a RAID 1+0 arrayprobably worse now that its a 2.7TB volume without raid (the only volume on that system that isn't mirrored.) Though wonder if I want to try zfs on linux again, or replace it with FreeBSD. I was faced with something like this on my Windows boxwhere eventually, I ended up writing off restoring from the local backup (a commercial time machine like product)...the mistake was using a windows fake raid5 external array as my backup drive. And, losing the system due to problems in the fake raid. I did briefly put together a CentOS live CD that could access the array, but the drives I copied the data to promptly failed on me shortly after I had broken the array and turned them into a raidz pool. Someday I need to get back to going through the disk image of the failed system drive and recover as much as possible from that. The box that was my Windows desktop is now my FreeBSD desktop Lawrence ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: backups using rsync
On 03/04/2013 05:35 AM, Ronald F. Guilmette wrote: > As a result of this past Black Friday weekend, I now enjoy a true abundance > of disk space, for the first time in my life. > > I wanna make a full backup, on a weekly basis, of my main system's shiny > new 1TB drive onto another 1TB drive that I also picked up cheap back on > Black Friday. > > I've been planning to set this up for some long time now, but I've only > gotten 'round to working on it now. > > Now, unfortunately, I have just been bitten by the evil... and apparently > widely known (except to me)... ``You can't use dump(8) to dump a journaled > filesystem with soft updates'' bug-a-boo. You can use dump(8) to dump a SU-journaled filesystem; you just cannot create a snapshot. This implies that dump(8) will be run against the live and possibly changing filesystem, which can lead to issues with the consistency of the contents of files thus dumped; but not necessarily with the consistency of the dump itself. Any tool that backs up a live filesystem, such as rsync or tar, will have these issues. > Sigh. The best laid plans of mice and men... > > I _had_ planned on using dump/restore and making backups from live mounted > filesystems while the system was running. But I really don't want to have > to take the system down to single-user mode every week for a few hours while > I'm making my disk-to-disk backup. So now I'm looking at doing the backups > using rsync. I've used rsync to back up Linux and FreeBSD machines daily for years, and I've never had a problem with the backups nor subsequent restorations. Especially for restorations of the laptop that ate SSDs. Having a decent snapshot capability on the backup target filesystem can help a lot if you want to maintain multiple sparse backup revisions; otherwise, you're stuck using creative scripting around rsync's --link-dest option. > I see that rsync can nowadays properly cope with all sorts of oddities, > like fer instance device files, hard-linked files, ACLs, file attributes, > and all sorts of other unusual but important filesystem thingies. That's > good news, but I still have to ask the obvious question: > > If I use all of the following rsync options... -a,-H,-A, -X, and -S > when trying to make my backups, and if I do whatever additional fiddling > is necessary to insure that I separately copy over the MBR and boot loader > also to my backup drive, then is there any reason that, in the event of > a sudden meteor shower that takes out my primary disk drive while leaving > my backup drive intact, I can't just unplug my old primary drive, plug in > my (rsync-created) backup drive, reboot and be back in the sadddle again, > almost immediately, and with -zero- problems? There will /always/ be problems. The best you can do is become familiar with the tools and procedures so you can tackle them when they happen. My suggestion for something that you can use as a warm standby is to create it as a warm standby: go through the entire installation procedure again for the backup drive, and then use rsync or suchlike to periodically synchronize the second filesystem with the first. When you update the boot code on one, do so on the other. Be extremely careful if you decide to do this with both disks attached to the same machine: if you use geom labels (gpt, ufs, glabel, et alia) or dynamically numbered storage devices, you can easily run into a situation where a reboot with both devices attached suddenly starts using your backup instead without you realizing it, or flips back and forth. > P.S. My apologies if I've already asked this exact same question here > before. I'm getting a sense of deja vu... or else a feeling that I am > often running around in circles, chasing my own tail. > > P.P.S. Before anyone asks, no I really _do not_ want to just use RAID > as my one and only backup strategy. RAID is swell if your only problem > is hardware failures. As far as I know however it will not save your > bacon in the event of a fumble fingers "rm -rf *" moment. Only frequent > and routine actual backups can do that. -- Fuzzy love, -CyberLeo Furry Peace! - http://www.fur.com/peace/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: backups using rsync
In message , Warren Block wrote: >Until SUJ has been deemed 100%, I avoid it and suggest others do also. >It can be disabled on an existing filesystem from single user mode. hehe Silly me! What do *I* know? I just go about my business and try not to create too much trouble for myself. To be honest and truthful I have to say that this journaling stuff entirely snuck up on me. I confess... I wasn't paying attention (to the world of FreeBSD innovations) and thus, when I moved myself recently to 9.x (from 8.3) I did so without even having been aware that the new filesystems that I was creating during my clean/fresh install of 9.1 had journaling turned on by default. (As the saying goes, I didn't get the memo.) Not that I mind, really. It sounds like a great concept and a great feature and I was happy to have it right up until the moment that "dump -L" told me to go pound sand. :-( So, um, I was reading about this last night, but I was sleepy and my eyes glazed over... Please remind me, what is the exact procedire for turning off the journaling? I boot to single user mode (from a live cd?) and then what? Is it tunefs with some special option? >> If I use all of the following rsync options... -a,-H,-A, -X, and -S >> when trying to make my backups, and if I do whatever additional fiddling >> is necessary to insure that I separately copy over the MBR and boot loader >> also to my backup drive, then is there any reason that, in the event of >> a sudden meteor shower that takes out my primary disk drive while leaving >> my backup drive intact, I can't just unplug my old primary drive, plug in >> my (rsync-created) backup drive, reboot and be back in the sadddle again, >> almost immediately, and with -zero- problems? > >It works. I use this to "slow mirror" SSDs to a hard disk, avoiding the >speed penalty of combining an SSD with a hard disk in RAID1. Great! Thanks Warren. >Use the latest net/rsync port, and enable the FLAGS option. I use these >options, copying each filesystem individually: > >-axHAXS --delete --fileflags --force-change Hummm... I guess that I have some non-current rsync installed. In the man page I have there is no mention of any "--force-change" option. What does it do? >Yes, the partitions and bootcode must be set up beforehand. After that, >it works. Good to know. Thanks again Warren. >Like any disk redundancy scheme, test it before an emergency. Naw. I like to live dangerously. :-) Regards, rfg ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: backups using rsync
In message <20130304125634.8450cfaf.free...@edvax.de>, Polytropon wrote: >On Mon, 04 Mar 2013 03:35:30 -0800, Ronald F. Guilmette wrote: >> Now, unfortunately, I have just been bitten by the evil... and apparently >> widely known (except to me)... ``You can't use dump(8) to dump a journaled >> filesystem with soft updates'' bug-a-boo. > >There are other tools you can use, for example tar or cpdup >or rsync, as you've mentioned in the subject. tar I already knew about, but I think you will agree that it has lots of limitations that make it entirely inappropriate for mirroring an entire system. This cpdup thing is entirely new to me. Thanks for mentioning it! I really never heard of it before, but I just now installed it from ports, and I'm perusing the man page. It looks very promising. Too bad it doesn't properly handle sparse files, but oh well. That's just a very minor nit. (Does it properly handle everything else that rsync claims to be able to properly handle, e.g. ACLs, file attributes, etc., etc.?) >The same problems that apply when dumping live systems can >bite you using rsync, What problems are we talking about, in particular? I am guessing that if I use rsync, then I *won't* encounter this rather annoying issue/problem relating to UFS filesystems that have both soft updates and journaling enabled, correct? >but support for this "on file system >level" seems to be better in rsync than what dump does "on >block level". What exactly did you mean by "this" ? >> If I use all of the following rsync options... -a,-H,-A, -X, and -S >> when trying to make my backups, and if I do whatever additional fiddling >> is necessary to insure that I separately copy over the MBR and boot loader >> also to my backup drive, then is there any reason that, in the event of >> a sudden meteor shower that takes out my primary disk drive while leaving >> my backup drive intact, I can't just unplug my old primary drive, plug in >> my (rsync-created) backup drive, reboot and be back in the sadddle again, >> almost immediately, and with -zero- problems? > >You would have to make sure _many_ things are consistent >on the backup disk. Well, this is what I am getting at. This is/was the whole point of my post and my question. I want to know: What is that set of things, exactly? >Regarding terminology, that would make the disk a failover disk OK. Thank you. I will henceforth use that terminology. >The disk would need to have an initialized file system and >a working boot mechanism, both things rsync does not deal with Check and check. I implicitly understood the former, and I explicitly mentioned the latter in my original post in this thread. But is there anything else, other than those two things (which, just as you say, are both clearly outside of the scope of what rsync does)? Anything else I need to do or worry about in order to be able to use rsync to create & maintain a full-blown fully-working system failover drive? If so, I'd much rather learn about it now... you know... as opposed to learning about it if and when I actually have to _use_ my failover drive. Regards, rfg ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: backups using rsync
On Mon, 4 Mar 2013, Ronald F. Guilmette wrote: Now, unfortunately, I have just been bitten by the evil... and apparently widely known (except to me)... ``You can't use dump(8) to dump a journaled filesystem with soft updates'' bug-a-boo. Until SUJ has been deemed 100%, I avoid it and suggest others do also. It can be disabled on an existing filesystem from single user mode. If I use all of the following rsync options... -a,-H,-A, -X, and -S when trying to make my backups, and if I do whatever additional fiddling is necessary to insure that I separately copy over the MBR and boot loader also to my backup drive, then is there any reason that, in the event of a sudden meteor shower that takes out my primary disk drive while leaving my backup drive intact, I can't just unplug my old primary drive, plug in my (rsync-created) backup drive, reboot and be back in the sadddle again, almost immediately, and with -zero- problems? It works. I use this to "slow mirror" SSDs to a hard disk, avoiding the speed penalty of combining an SSD with a hard disk in RAID1. Use the latest net/rsync port, and enable the FLAGS option. I use these options, copying each filesystem individually: -axHAXS --delete --fileflags --force-change --delete removes files present on the copy that are not on the original. Some people may want to leave those. --exclude= is used on certain filesystems to skip directories that are full of easily recreated data that changes often, like /usr/obj. Yes, the partitions and bootcode must be set up beforehand. After that, it works. Like any disk redundancy scheme, test it before an emergency. P.P.S. Before anyone asks, no I really _do not_ want to just use RAID as my one and only backup strategy. RAID is swell if your only problem is hardware failures. As far as I know however it will not save your bacon in the event of a fumble fingers "rm -rf *" moment. Only frequent and routine actual backups can do that. Yes, RAID is not a backup. Another suggestion I've been making often: use sysutils/rsnapshot to make an accessible history of files. The archive go on another partition on the mirror drive, which likely has more space than the original. rsnapshot uses rsync with hard links to make an archive that lets you easily get to old versions of files that have changed in the last few hours/days/weeks/months. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: backups using rsync
On Mon, 04 Mar 2013 03:35:30 -0800, Ronald F. Guilmette wrote: > Now, unfortunately, I have just been bitten by the evil... and apparently > widely known (except to me)... ``You can't use dump(8) to dump a journaled > filesystem with soft updates'' bug-a-boo. There are other tools you can use, for example tar or cpdup or rsync, as you've mentioned in the subject. > I _had_ planned on using dump/restore and making backups from live mounted > filesystems while the system was running. But I really don't want to have > to take the system down to single-user mode every week for a few hours while > I'm making my disk-to-disk backup. So now I'm looking at doing the backups > using rsync. The same problems that apply when dumping live systems can bite you using rsync, but support for this "on file system level" seems to be better in rsync than what dump does "on block level". > If I use all of the following rsync options... -a,-H,-A, -X, and -S > when trying to make my backups, and if I do whatever additional fiddling > is necessary to insure that I separately copy over the MBR and boot loader > also to my backup drive, then is there any reason that, in the event of > a sudden meteor shower that takes out my primary disk drive while leaving > my backup drive intact, I can't just unplug my old primary drive, plug in > my (rsync-created) backup drive, reboot and be back in the sadddle again, > almost immediately, and with -zero- problems? You would have to make sure _many_ things are consistent on the backup disk. Regarding terminology, that would make the disk a failover disk, even if the act of making it the actual "work disk" is something you do manually. The disk would need to have an initialized file system and a working boot mechanism, both things rsync does not deal with, if I remember correctly. But as soon as you have initialized the disk for the first time and made sure (by testing your first result of a rsync run), it should work with any subsequent change of data you transfer to that disk. > P.P.S. Before anyone asks, no I really _do not_ want to just use RAID > as my one and only backup strategy. RAID _is_ **NO** backup. It's for dedundancy and performance. If something is erased or corrupted, it's on all disks. And all the disks permanently run. A backup disk only runs twice: when backing something up, or when restoring. In your case, restoring means that the disk is put into operation in its role as a failover disk. > RAID is swell if your only problem > is hardware failures. Still hardware failures can corrupt data on all participating disks. > As far as I know however it will not save your > bacon in the event of a fumble fingers "rm -rf *" moment. Only frequent > and routine actual backups can do that. Correct. It's important to learn that lesson _before_ it is actually needed. :-) -- Polytropon Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
backups using rsync
As a result of this past Black Friday weekend, I now enjoy a true abundance of disk space, for the first time in my life. I wanna make a full backup, on a weekly basis, of my main system's shiny new 1TB drive onto another 1TB drive that I also picked up cheap back on Black Friday. I've been planning to set this up for some long time now, but I've only gotten 'round to working on it now. Now, unfortunately, I have just been bitten by the evil... and apparently widely known (except to me)... ``You can't use dump(8) to dump a journaled filesystem with soft updates'' bug-a-boo. Sigh. The best laid plans of mice and men... I _had_ planned on using dump/restore and making backups from live mounted filesystems while the system was running. But I really don't want to have to take the system down to single-user mode every week for a few hours while I'm making my disk-to-disk backup. So now I'm looking at doing the backups using rsync. I see that rsync can nowadays properly cope with all sorts of oddities, like fer instance device files, hard-linked files, ACLs, file attributes, and all sorts of other unusual but important filesystem thingies. That's good news, but I still have to ask the obvious question: If I use all of the following rsync options... -a,-H,-A, -X, and -S when trying to make my backups, and if I do whatever additional fiddling is necessary to insure that I separately copy over the MBR and boot loader also to my backup drive, then is there any reason that, in the event of a sudden meteor shower that takes out my primary disk drive while leaving my backup drive intact, I can't just unplug my old primary drive, plug in my (rsync-created) backup drive, reboot and be back in the sadddle again, almost immediately, and with -zero- problems? Regards, rfg P.S. My apologies if I've already asked this exact same question here before. I'm getting a sense of deja vu... or else a feeling that I am often running around in circles, chasing my own tail. P.P.S. Before anyone asks, no I really _do not_ want to just use RAID as my one and only backup strategy. RAID is swell if your only problem is hardware failures. As far as I know however it will not save your bacon in the event of a fumble fingers "rm -rf *" moment. Only frequent and routine actual backups can do that. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Syncing Two Dirs With Rsync
On 2013-01-10 12:57, Tim Daneliuk wrote: I have used rsync for many years to make sure a destination machine:directory is kept up-to-date with some source master directory. I now need to find a way to keep two different machine:dirs in sync with each other. But for any given file, I don't know which of these is newer so I don't know "which way" to sync. For example given: machineA::/dir/foo machineB:/dir/foo machineA::/dir/bar machineB:/dir/bar Say the machineA has the newest foo, but machineB has the newest bar. At the end of syncing, I want both machines to have the latest copies of everything. I'm guessing there's a way to do this with rsync but I'm kind of stumped. Ideas? Never tried it before, but I would think you could use the -u option to skip newer files, and do a push followed by a pull so that the newer files from each side get copied. However you wouldn't be able to use the delete files option as new files could get deleted instead of copied, depending on which side ran first. -- Thanks, Dean E. Weimer http://www.dweimer.net/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Syncing Two Dirs With Rsync
On Jan 10, 2013, at 10:57 AM, Tim Daneliuk wrote: > I have used rsync for many years to make sure a destination > machine:directory is kept up-to-date with some source master > directory. > > I now need to find a way to keep two different machine:dirs > in sync with each other. But for any given file, I don't know > which of these is newer so I don't know "which way" to sync. > > For example given: > > machineA::/dir/foo machineB:/dir/foo > machineA::/dir/bar machineB:/dir/bar > > Say the machineA has the newest foo, but machineB has the > newest bar. At the end of syncing, I want both machines > to have the latest copies of everything. > > I'm guessing there's a way to do this with rsync but I'm kind > of stumped. rsync's --update flag will not overwrite a file at the destination if it was modified more recently then the source location. So you can run rsync twice to sync from A to B and then from B to A. Make very sure both boxes are keeping correct time and/or are mutually sync'ed via NTP or similar. However, if you make different changes to the same file on A and on B, you will lose one of them. (That is what version control systems like SVN and git would resolve. So if you do plan to do 2-way or N-way changes and sync'ing on a regular basis, version control is much less likely to lose changes or otherwise screw up.) Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Syncing Two Dirs With Rsync
I have used rsync for many years to make sure a destination machine:directory is kept up-to-date with some source master directory. I now need to find a way to keep two different machine:dirs in sync with each other. But for any given file, I don't know which of these is newer so I don't know "which way" to sync. For example given: machineA::/dir/foo machineB:/dir/foo machineA::/dir/bar machineB:/dir/bar Say the machineA has the newest foo, but machineB has the newest bar. At the end of syncing, I want both machines to have the latest copies of everything. I'm guessing there's a way to do this with rsync but I'm kind of stumped. Ideas? -- --- Tim Daneliuk ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Backup with mtree and rsync?
No (not directly, except overwriting directories with content), but cpdup can; see "man cpdup" for details and inspiration. True, but cpdup is not part of the base system. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Backup with mtree and rsync?
I apparently reinvented the wheel. :-) Thanks for the link, it is indeed very inspiring. Quoting Ciprian Dorin Craciun : On Sat, Jan 5, 2013 at 8:12 PM, wrote: I have been wondering whether it is possible to create a backup system using mtree and rsync. Essentially, the user would create a mtree specification of the source directory and copy it over to the destination directory with rsync. Any changes in the destination could then be detected before restoring with the mtree specification, which should contain strong hashes of the files and should not contain the nlink keyword. A little bit off-topic, but there is a small tool that does something similar to your suggested `mtree` usage, but specifically tailored for backups, `rdup`: http://miek.nl/projects/rdup Although I've not used it myself (I use `rdiff-backup` and on Linux), the idea is pretty similar with what you want to achieve: * you run `rdup` with an old "descriptor file" plus a target path, and in turn it generates: * a new "descriptor file"; * a list of files that should be backed up; * you then decide what you do with the list of files to be backed-up (i.e. put them in a `tar`, `rysnc` them to a server, etc.); Hope it helps, Ciprian. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Backup with mtree and rsync?
On Sat, Jan 5, 2013 at 8:12 PM, wrote: > I have been wondering whether it is possible to create a backup system > using mtree and rsync. Essentially, the user would create a mtree > specification of the source directory and copy it over to the destination > directory with rsync. Any changes in the destination could then be > detected before restoring with the mtree specification, which should > contain strong hashes of the files and should not contain the nlink > keyword. A little bit off-topic, but there is a small tool that does something similar to your suggested `mtree` usage, but specifically tailored for backups, `rdup`: http://miek.nl/projects/rdup Although I've not used it myself (I use `rdiff-backup` and on Linux), the idea is pretty similar with what you want to achieve: * you run `rdup` with an old "descriptor file" plus a target path, and in turn it generates: * a new "descriptor file"; * a list of files that should be backed up; * you then decide what you do with the list of files to be backed-up (i.e. put them in a `tar`, `rysnc` them to a server, etc.); Hope it helps, Ciprian. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Backup with mtree and rsync?
On Tue, 08 Jan 2013 15:57:39 -0200, schu...@ime.usp.br wrote: > > It's possible that the mtree support in tar(8) might be able to do it, > > but it would probably be a lot slower. > > Wait, can tar be used to remove files? No (not directly, except overwriting directories with content), but cpdup can; see "man cpdup" for details and inspiration. -- Polytropon Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Backup with mtree and rsync?
I don't see any way to do this directly. What you probably want to do is use find(1) to pick out the new files to check, and then merge the changes into the old mtree(8) spec. Not trivial, but the spec syntax is intended to be easy to parse, so it shouldn't be that hard either. What I am currently doing somewhat fits your description. I feed find output into a C program that merges the old description with the directory state to produce a new description. However, I use a format different than mtree. I was seeking a shorter, more elegant, solution. It's possible that the mtree support in tar(8) might be able to do it, but it would probably be a lot slower. Wait, can tar be used to remove files? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Backup with mtree and rsync?
schu...@ime.usp.br writes: > I have been wondering whether it is possible to create a backup system > using mtree and rsync. Essentially, the user would create a mtree > specification of the source directory and copy it over to the destination > directory with rsync. Any changes in the destination could then be > detected before restoring with the mtree specification, which should > contain strong hashes of the files and should not contain the nlink > keyword. > > The problem is that mtree would be too slow. It would recompute the > hashes of big files even when they did not change from the last backup. > Therefore, I would like to ask if there is an easy way to accomplish > the following. > > Let a mtree specification of a directory from a certain point in the > past be given. Also, assume that a (regular) file below that directory > has not changed if its current modification time (mtime) equals > its modification time in the past specification. > Produce as output the new mtree specification for the directory without > reading these files. > > This is somewhat like rsync does to perform incremental backups. Except that you have a spec for mtree to be sure the backup copy hasn't been corrupted. I don't see any way to do this directly. What you probably want to do is use find(1) to pick out the new files to check, and then merge the changes into the old mtree(8) spec. Not trivial, but the spec syntax is intended to be easy to parse, so it shouldn't be that hard either. > P.S.: As an aside, is there an utility in the base system that can > reproduce the behavior of `rsync --delete -a dir0/ dir1/`? It's possible that the mtree support in tar(8) might be able to do it, but it would probably be a lot slower. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Backup with mtree and rsync?
I have been wondering whether it is possible to create a backup system using mtree and rsync. Essentially, the user would create a mtree specification of the source directory and copy it over to the destination directory with rsync. Any changes in the destination could then be detected before restoring with the mtree specification, which should contain strong hashes of the files and should not contain the nlink keyword. The problem is that mtree would be too slow. It would recompute the hashes of big files even when they did not change from the last backup. Therefore, I would like to ask if there is an easy way to accomplish the following. Let a mtree specification of a directory from a certain point in the past be given. Also, assume that a (regular) file below that directory has not changed if its current modification time (mtime) equals its modification time in the past specification. Produce as output the new mtree specification for the directory without reading these files. This is somewhat like rsync does to perform incremental backups. P.S.: As an aside, is there an utility in the base system that can reproduce the behavior of `rsync --delete -a dir0/ dir1/`? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Rsync and Preservation of Ownership and Permissions
Michael Sierchio writes: > Does the same user exist on the remote system, with the same uid, etc.? Yes. > If you're using rsync with ssh as the transport, and connecting to the > remote machine as the backups user, that's who will own the files on > its local filesystem... I thought rsync had some encoding it might slip in to the tree that another rsync run as root on the recovering system could use to figure out all those thousands of ownerships and get them all straight, but this makes perfect sense. > You've written a lot of narrative, but show us precisely what commands > you're running. Why would you run the command as root, and ssh as > backups, when you want them to be owned by "normal" ? Because root is the only user who can "see" files from all other users so root starts the process. Here is what I tried. Remember, folks, this will not work! This tries to backup a system named z. ##!/bin/sh #rsync --delete -alHvq --exclude "/proc" // back...@backup-server.okstate.edu:z > You can run the command as root, and use restricted ssh keys (use > authorized_keys to restrict it to executing a specific rsync > command) you can run rsync as a regular user to that user's > account on the remote system... per...@pluto.rain.com writes: > Perhaps you could have rsync log in to a jail on the backup server, > where it could safely be granted root permission. Hmm. It's all rather clear, now. A jailed environment that looks like root is about the only thing that could work. Thank you. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Rsync and Preservation of Ownership and Permissions
Martin McCormick wrote: > Rsync is a great utility, but is there a way to preserve > ownership and permissions if rsync remotely logs in to a backup > server as a normal user? AFAIK, no, because only root may change the ownership of a file -- see chown(2). > Any ideas are greatly appreciated. Perhaps you could have rsync log in to a jail on the backup server, where it could safely be granted root permission. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Rsync and Preservation of Ownership and Permissions
On Wed, Nov 23, 2011 at 9:01 PM, Martin McCormick wrote: > Rsync is a great utility, but is there a way to preserve > ownership and permissions if rsync remotely logs in to a backup > server as a normal user? Does the same user exist on the remote system, with the same uid, etc.? If you're using rsync with ssh as the transport, and connecting to the remote machine as the backups user, that's who will own the files on its local filesystem... You've written a lot of narrative, but show us precisely what commands you're running. Why would you run the command as root, and ssh as backups, when you want them to be owned by "normal" ? You can run the command as root, and use restricted ssh keys (use authorized_keys to restrict it to executing a specific rsync command) you can run rsync as a regular user to that user's account on the remote system... > The recovery process is run by root but copies all the > files from the backup server as a normal user and uses its root > capabilities to restore them. > > What happens now is that all the files end up owned by > and in the group of the user ID that copied the information from > the client to the server. That's obviously not too useful so I > suspect there is a better way than trying to make a remote login > to root from another system. > > Basically, cron starts a backup as root on system A. > System A makes a remote ssh connection using the -e flag to > backups@server. The system trying to recover the files starts a > rsync process as root which remotely connects to backups@server > to retrieve the files. > > In practice, the files come across but every last one of > them is owned by and in the group of user backups. > > Any ideas are greatly appreciated. > > Thank you. > ___ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org" > ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Rsync and Preservation of Ownership and Permissions
Rsync is a great utility, but is there a way to preserve ownership and permissions if rsync remotely logs in to a backup server as a normal user? The recovery process is run by root but copies all the files from the backup server as a normal user and uses its root capabilities to restore them. What happens now is that all the files end up owned by and in the group of the user ID that copied the information from the client to the server. That's obviously not too useful so I suspect there is a better way than trying to make a remote login to root from another system. Basically, cron starts a backup as root on system A. System A makes a remote ssh connection using the -e flag to backups@server. The system trying to recover the files starts a rsync process as root which remotely connects to backups@server to retrieve the files. In practice, the files come across but every last one of them is owned by and in the group of user backups. Any ideas are greatly appreciated. Thank you. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync and the ports tree
Sounds like a well-thought out backup strategy. I've started to use your methods here, and I'm building ports I need at the same time, but the wireless here is not password protected so is incredibly slow as there are lots of leaches on the system. But while I'm plodding along, I have a few more questions: On Wed, Oct 26, 2011 at 8:07 AM, C. P. Ghost wrote: > On Wed, Oct 26, 2011 at 12:39 PM, Peter Kryszkiewicz > wrote: > > I have several machines installed in my temporary location and only my > > laptop gets the internet through wireless. So far I've been building > ports > > on the other machines by rsync'ing the distfiles from the laptop as I > need > > them (all machines have the same FreeBSD 8.2 installed). > > > > The problem comes after I did a 'portupgrade -a' on the laptop. To ensure > > the other ports trees are in sync, can I rsync the /usr/ports directory > to > > the other machines? Since some of them are different architectures (amd64 > > multicore for instance) I ran into situations where the distfiles are > > different (for gcc for example). > > First of all, rsync is working perfectly if you want to > distribute /usr/ports/distfiles, /usr/ports to your internal > machines, even when they are not of the same architecture. > I'm doing this with a BIG farm of servers running i386, amd64, > and sparc64 for a long, long time. > > You only need to make sure to rsync the *union* of your > /usr/ports/distfiles directories, or else it won't work. > > Say, on amd64 you have > /usr/ports/distfiles/some-distfile-for-amd64-only.tar.bz2 > and on i386 you have > /usr/ports/distfiles/some-distfile-for-i386-only.tar.bz2 > > Yes, that happens every now and then. > > So you have to rsync both ways, so that you end up with > /usr/ports/distfiles/some-distfile-for-amd64-only.tar.bz2 > /usr/ports/distfiles/some-distfile-for-i386-only.tar.bz2 > on both i386 and amd64 machines. > I've done that, it works well especially for the architecture differences. gcc requires an additional distfile for the amd64 build). > > The catch is: look out for rsync's --delete flag! When some > port managers delete old/stale distfiles, they may also delete > distfiles for the *other* arches because they (rightly) think > they are not needed here... and when you then rsync with --delete, > that would (wrongly) propagate such deletes to those arches, > and you end up with missing distfiles on the targets. > > Since I have more than just two arches, I use a slightly different > 2-layer workflow: > > 0. I have 3 servers that are allowed to fetch files from the outside: > i386-master, amd64-master, sparc64-master. > and a whole bunch of i386-slave-NNN, amd64-slave-NNN and > sparc64-slave-NNN machines that would duplicate from their > relative masters via rsync. > > On all -master(s), I keep $DISTFILES outside of /usr/ports > (on /usr/local/distfiles, with a symbolic link in /usr/ports > /usr/ports/distfiles -> /usr/local/distfiles) > > Initial update of i386-master, as usual: > > 1. On i386-master, csup /usr/ports. > Run portmaster as usual to upgrade everything. > This may delete old stale distfiles and non-i386-distfiles. > This may fetch additional generic and i386-specific distfiles. > > Copy the new /usr/ports (without distfiles) to the other > arch masters: > > 2. rsync -av --delete i386-master:/usr/ports to amd64-master > and sparc64-master. CAUTION: Use --delete is okay, but only > because distfiles are not under /usr/ports, so as not to nuke > non-i386-specific distfiles of the other arches. > > Copy i386-master's NEW distfiles to the other arch masters: > > 3. rsync -av i386-master:/usr/local/distfiles to amd64-master > and sparc64-master. BEWARE: Don't use --delete here! > Do this to copy new generic distfiles (and i386) from > the i386-master build to amd64-master and sparc64-master. > > Update amd64-master and sparc64-master's ports as usual: > > 4. On amd64-master, run portmaster as usual to upgrade everything. > This may delete old stale distfiles and non-amd64-distfiles. > This may fetch additional (generic and) amd64-specific-distfiles. > > 5. On sparc64-master, run portmaster as usual to upgrade everything. > This may delete old stale distfiles and non-sparc64-distfiles. > This may fetch additional (generic and) sparc64-specific-distfiles. > > At this point, i386-master, amd64-master and sparc64-master are > fully updated, and their /usr/local/distfiles directories are up > to date w.r.t. their specific architectures. Now, copy everything > from the masters to the slaves: > >
Re: rsync and the ports tree
On Wed, Oct 26, 2011 at 12:39 PM, Peter Kryszkiewicz
wrote:
> I have several machines installed in my temporary location and only my
> laptop gets the internet through wireless. So far I've been building ports
> on the other machines by rsync'ing the distfiles from the laptop as I need
> them (all machines have the same FreeBSD 8.2 installed).
>
> The problem comes after I did a 'portupgrade -a' on the laptop. To ensure
> the other ports trees are in sync, can I rsync the /usr/ports directory to
> the other machines? Since some of them are different architectures (amd64
> multicore for instance) I ran into situations where the distfiles are
> different (for gcc for example).
First of all, rsync is working perfectly if you want to
distribute /usr/ports/distfiles, /usr/ports to your internal
machines, even when they are not of the same architecture.
I'm doing this with a BIG farm of servers running i386, amd64,
and sparc64 for a long, long time.
You only need to make sure to rsync the *union* of your
/usr/ports/distfiles directories, or else it won't work.
Say, on amd64 you have
/usr/ports/distfiles/some-distfile-for-amd64-only.tar.bz2
and on i386 you have
/usr/ports/distfiles/some-distfile-for-i386-only.tar.bz2
Yes, that happens every now and then.
So you have to rsync both ways, so that you end up with
/usr/ports/distfiles/some-distfile-for-amd64-only.tar.bz2
/usr/ports/distfiles/some-distfile-for-i386-only.tar.bz2
on both i386 and amd64 machines.
The catch is: look out for rsync's --delete flag! When some
port managers delete old/stale distfiles, they may also delete
distfiles for the *other* arches because they (rightly) think
they are not needed here... and when you then rsync with --delete,
that would (wrongly) propagate such deletes to those arches,
and you end up with missing distfiles on the targets.
Since I have more than just two arches, I use a slightly different
2-layer workflow:
0. I have 3 servers that are allowed to fetch files from the outside:
i386-master, amd64-master, sparc64-master.
and a whole bunch of i386-slave-NNN, amd64-slave-NNN and
sparc64-slave-NNN machines that would duplicate from their
relative masters via rsync.
On all -master(s), I keep $DISTFILES outside of /usr/ports
(on /usr/local/distfiles, with a symbolic link in /usr/ports
/usr/ports/distfiles -> /usr/local/distfiles)
Initial update of i386-master, as usual:
1. On i386-master, csup /usr/ports.
Run portmaster as usual to upgrade everything.
This may delete old stale distfiles and non-i386-distfiles.
This may fetch additional generic and i386-specific distfiles.
Copy the new /usr/ports (without distfiles) to the other
arch masters:
2. rsync -av --delete i386-master:/usr/ports to amd64-master
and sparc64-master. CAUTION: Use --delete is okay, but only
because distfiles are not under /usr/ports, so as not to nuke
non-i386-specific distfiles of the other arches.
Copy i386-master's NEW distfiles to the other arch masters:
3. rsync -av i386-master:/usr/local/distfiles to amd64-master
and sparc64-master. BEWARE: Don't use --delete here!
Do this to copy new generic distfiles (and i386) from
the i386-master build to amd64-master and sparc64-master.
Update amd64-master and sparc64-master's ports as usual:
4. On amd64-master, run portmaster as usual to upgrade everything.
This may delete old stale distfiles and non-amd64-distfiles.
This may fetch additional (generic and) amd64-specific-distfiles.
5. On sparc64-master, run portmaster as usual to upgrade everything.
This may delete old stale distfiles and non-sparc64-distfiles.
This may fetch additional (generic and) sparc64-specific-distfiles.
At this point, i386-master, amd64-master and sparc64-master are
fully updated, and their /usr/local/distfiles directories are up
to date w.r.t. their specific architectures. Now, copy everything
from the masters to the slaves:
6. On every i386-slave-NNN, rsync -av --delete:
/usr/ports, /usr/local (including /usr/local/distfiles),
/var/db/pkg, /var/db/ports
from i386-master.
7. On every amd64-slave-NNN, rsync -av --delete:
/usr/ports, /usr/local (including /usr/local/distfiles)
/var/db/pkg, /var/db/ports
from amd64-master.
8. On every sparc64-slave-NNN, rsync -av --delete:
/usr/ports, /usr/local (including /usr/local/distfiles)
/var/db/pkg, /var/db/ports
from sparc64-master.
You may also need to update entries in /etc and /usr/local/etc
on the slaves.
> If not rsync, what is the best way to keep multiple ports trees on different
> hardware in sync, assuming everything runs FreeBSD 8.2?
>
> regards,
> Peter Kryszkiewicz
-cpghost.
--
Cordula's Web. http://www.cordula.ws/
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync and the ports tree
On Wed, Oct 26, 2011 at 12:39 PM, Peter Kryszkiewicz wrote: > I have several machines installed in my temporary location and only my > laptop gets the internet through wireless. So far I've been building ports > on the other machines by rsync'ing the distfiles from the laptop as I need > them (all machines have the same FreeBSD 8.2 installed). > > The problem comes after I did a 'portupgrade -a' on the laptop. To ensure > the other ports trees are in sync, can I rsync the /usr/ports directory to > the other machines? Since some of them are different architectures (amd64 > multicore for instance) I ran into situations where the distfiles are > different (for gcc for example). > The distfiles are not different between architectures. Rsyncing /usr/ports works fine. But if you will bump into problems if you also sync /usr/ports/packages and you have different archs (i386 vs amd64 for instance). -- chs, ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
rsync and the ports tree
I have several machines installed in my temporary location and only my laptop gets the internet through wireless. So far I've been building ports on the other machines by rsync'ing the distfiles from the laptop as I need them (all machines have the same FreeBSD 8.2 installed). The problem comes after I did a 'portupgrade -a' on the laptop. To ensure the other ports trees are in sync, can I rsync the /usr/ports directory to the other machines? Since some of them are different architectures (amd64 multicore for instance) I ran into situations where the distfiles are different (for gcc for example). If not rsync, what is the best way to keep multiple ports trees on different hardware in sync, assuming everything runs FreeBSD 8.2? regards, Peter Kryszkiewicz ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync over nfs or rsync protocol
On Sun, Sep 25, 2011 at 09:10:22AM -0700, Jason C. Wells wrote: > On 09/23/11 14:11, Jerry McAllister wrote: > >Why would you interject NFS in the middle of it? jerry > There would be no middle. I would run rsyncd or nfsd, but not both. Ah, I get it. In that case, I think rsync is probably more useful than NFS because it can check and only copy modified files. Alternatively, if you are doing backups to recover from system failures - such as a disk crash, you would probably prefer dump(8)/restore(8) They can write to a file on the other machine, can do "change dumps" and they preserve all the needed UNIX attributes. I actually do a dump piped to a restore on another disk as a convenient backup to handle my all too frequent cases of fumble fingers and sleep deprived bad thinking where I need to quickly get back a file I mangled, deleted or need to start over on. Restore can easily pull single files or directory trees from a dump file as well. But having it already pre-restored makes it easier -- and only doubles my disk use - disk is cheap isn't it. jerry > > Jason ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync over nfs or rsync protocol
On 09/23/11 14:15, Chuck Swiger wrote: Lots. The handbook has a chapter on backups which is worth reading, also Regards, Ah the handbook. I forgot all about it. Thanks. Later, Jason ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync over nfs or rsync protocol
On 09/23/11 14:11, Jerry McAllister wrote: Why would you interject NFS in the middle of it? jerry There would be no middle. I would run rsyncd or nfsd, but not both. Jason ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync over nfs or rsync protocol
On Fri, Sep 23, 2011 at 2:15 PM, Chuck Swiger wrote: > Lots. The handbook has a chapter on backups which is worth reading, also True. Personally, I like dump/restore for disaster-recovery backups on FreeBSD. However, if you frequently need individual file recovery (e.g., Joe User deletes an important file and wants it back) or if you need to back up systems running something other than FreeBSD, BackupPC is hard to beat for disk-to-disk backup. If you're going to tape you might want something like Amanda, instead. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync over nfs or rsync protocol
On Fri, Sep 23, 2011, Jason C. Wells wrote: > I am looking into finally setting up a backup solution that's a little > more sophisticated than a bunch of DVD-RWs. I have two servers. I'd > like to make each a backup server for the other. I'm considering using > rsync. > > Is rsync a good choice for a backup tool? > Should I use the rsyncd or should I use NFS? I'm using 100 mbps ethernet. > What's the better solution I haven't considered? We use rsync extensively for backups on local LANs as well as off-site via the 'Net. My experience using NFS for this was not good, but it's been many years since I last tried this so I don't remember the details. Bill -- INTERNET: b...@celestial.com Bill Campbell; Celestial Software LLC URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way Voice: (206) 236-1676 Mercer Island, WA 98040-0820 Fax:(206) 232-9186 Skype: jwccsllc (206) 855-5792 Windows is a computer virus with a user interface!! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync over nfs or rsync protocol
On Sep 23, 2011, at 2:04 PM, Jason C. Wells wrote: > Is rsync a good choice for a backup tool? It's OK. A versioned backup system (dump/restore, Legato Networker, Amanda, Retrospect, etc) is more efficient at using backup storage. > Should I use the rsyncd or should I use NFS? I'm using 100 mbps ethernet. If it's local and you already have NFS in place, that would be fine. If you're backing up over a WAN, rsyncd is probably a better call. > What's the better solution I haven't considered? Lots. The handbook has a chapter on backups which is worth reading, also Regards, -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync over nfs or rsync protocol
On Fri, Sep 23, 2011 at 2:04 PM, Jason C. Wells wrote: > I am looking into finally setting up a backup solution that's a little more > sophisticated than a bunch of DVD-RWs. I have two servers. I'd like to > make each a backup server for the other. I'm considering using rsync. > > Is rsync a good choice for a backup tool? rsync is good for mirroring one file tree to another. If you want a complete backup system centering around rsync, check out BackupPC. NFS will let you mount one server's files from another, but you'll still need to use something to copy the files over; either rsync or something else. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync over nfs or rsync protocol
On Fri, Sep 23, 2011 at 02:04:03PM -0700, Jason C. Wells wrote: > I am looking into finally setting up a backup solution that's a little > more sophisticated than a bunch of DVD-RWs. I have two servers. I'd > like to make each a backup server for the other. I'm considering using > rsync. > Is rsync a good choice for a backup tool? rsync can be used effectively. The optimum solution really depends on how much total disk and how much changes between each run. > Should I use the rsyncd or should I use NFS? I'm using 100 mbps ethernet. > What's the better solution I haven't considered? Why would you interject NFS in the middle of it? jerry > > Thanks, > Jason > ___ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org" ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
RE: rsync over nfs or rsync protocol
I'm sure you'll get a TON of responses on this. Maybe some script that combines tar, g[b]zip, and rsync? I wouldn't use NFS just for this, but if it's already there it may have a place. -Original Message- From: owner-freebsd-questi...@freebsd.org [mailto:owner-freebsd-questi...@freebsd.org] On Behalf Of Jason C. Wells Sent: Friday, September 23, 2011 4:04 PM To: freebsd general questions Subject: rsync over nfs or rsync protocol I am looking into finally setting up a backup solution that's a little more sophisticated than a bunch of DVD-RWs. I have two servers. I'd like to make each a backup server for the other. I'm considering using rsync. Is rsync a good choice for a backup tool? Should I use the rsyncd or should I use NFS? I'm using 100 mbps ethernet. What's the better solution I haven't considered? Thanks, Jason ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org" "This email is intended to be reviewed by only the intended recipient and may contain information that is privileged and/or confidential. If you are not the intended recipient, you are hereby notified that any review, use, dissemination, disclosure or copying of this email and its attachments, if any, is strictly prohibited. If you have received this email in error, please immediately notify the sender by return email and delete this email from your system." ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
rsync over nfs or rsync protocol
I am looking into finally setting up a backup solution that's a little more sophisticated than a bunch of DVD-RWs. I have two servers. I'd like to make each a backup server for the other. I'm considering using rsync. Is rsync a good choice for a backup tool? Should I use the rsyncd or should I use NFS? I'm using 100 mbps ethernet. What's the better solution I haven't considered? Thanks, Jason ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
rsync connection to Windows 2008 Server
I have run into a problem that I cannot find the answer to using Google and I am hoping someone has seen this before. I am connecting to a Windows 2008 server running cwRsync as a daemon. One of my six sites will not connect to the server. No error messages are seen in the cwRsync log file or the Event Viewer on the Windows 2008 computer. My network configuration is as follows. STL --WAN-- Firewall --IPsec Tunnel -- 192.168.100.56. On the remote computer, using the following command line, and I receive the following messages. rsync -avz /home 192.168.100.56::Homes rsync: read error: Connection reset by peer (54) rsync error: error in rsync protocol data stream (code 12) at io.c(604) [sender=2.6.9] Looking at tcpdump, I think the connection is started, but the computer on my end is not responding after a certain time. 12:00:17.817759 IP (tos 0x0, ttl 64, id 64973, offset 0, flags [DF], proto: TCP (6), length: 64) N-ST-LOUIS.65460 > 192.168.100.56.rsync: S, cksum 0x9834 (correct), 1279331061:1279331061(0) win 65535 12:00:17.817891 IP (tos 0x0, ttl 64, id 64973, offset 0, flags [DF], proto: TCP (6), length: 64) N-ST-LOUIS.65460 > 192.168.100.56.rsync: S, cksum 0x9834 (correct), 1279331061:1279331061(0) win 65535 12:00:17.871976 IP (tos 0x0, ttl 63, id 11982, offset 0, flags [none], proto: TCP (6), length: 52) 192.168.100.56.rsync > N-ST-LOUIS.65460: S, cksum 0xb379 (correct), 992395985:992395985(0) ack 1279331062 win 5640 12:00:17.872064 IP (tos 0x0, ttl 64, id 64974, offset 0, flags [DF], proto: TCP (6), length: 40) N-ST-LOUIS.65460 > 192.168.100.56.rsync: ., cksum 0x889c (correct), 1:1(0) ack 1 win 33120 12:00:17.872233 IP (tos 0x0, ttl 64, id 64974, offset 0, flags [DF], proto: TCP (6), length: 40) N-ST-LOUIS.65460 > 192.168.100.56.rsync: ., cksum 0x889c (correct), 1:1(0) ack 1 win 33120 12:00:17.872280 IP (tos 0x0, ttl 64, id 64975, offset 0, flags [DF], proto: TCP (6), length: 52) N-ST-LOUIS.65460 > 192.168.100.56.rsync: P, cksum 0x0923 (correct), 1:13(12) ack 1 win 33120 12:00:17.872349 IP (tos 0x0, ttl 64, id 64975, offset 0, flags [DF], proto: TCP (6), length: 52) N-ST-LOUIS.65460 > 192.168.100.56.rsync: P, cksum 0x0923 (correct), 1:13(12) ack 1 win 33120 12:00:17.872592 IP (tos 0x0, ttl 63, id 11984, offset 0, flags [none], proto: TCP (6), length: 40) 192.168.100.56.rsync > N-ST-LOUIS.65460: ., cksum 0x0f00 (correct), 1:1(0) ack 13 win 64240 12:00:38.882477 IP (tos 0x0, ttl 63, id 12123, offset 0, flags [none], proto: TCP (6), length: 40) 192.168.100.56.rsync > N-ST-LOUIS.65460: R, cksum 0x4848 (correct), 992395986:992395986(0) win 4344 For testing, my firewall rules were set to pass in quick all, pass out quick all. The remote server is at the end of an ipsec tunnel maintained by racoon. Ping times are reasonable 53ms between sites. What really baffles me is the other five (5) sites I maintain are connecting without any problems. All are running the same version of rsync. Any suggestions would be greatly appreciated. Thanks, Jay ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Any way to have login output AND use rsync/scp ?
On Mon, 2 May 2011 02:13:17 +0200, Polytropon wrote: > On Sun, 1 May 2011 15:58:45 -0700 (PDT), George Sanders > wrote: > > I have my .cshrc file run some basic netstat and 'w' commands so that when > > I log > > in, I can see at a glance what is going on on the system and notice any > > unusual > > login activity, etc. > > > > However this completely breaks both scp and rsync - they cannot function at > > all > > (apparently) with any kind of stdio output from the shell. > > > > Is there any way around this ? > > > Create a file ~/.login and put your commands (in sh syntax, > not csh) there. This file will only be executed at interactive > logins. See "man csh", section FILES for details. Sorry, wrote before thinking. :-) Of course ~/.login is csh syntax, as I would guess after (missed to) read the file's header which states: # .login - csh login script, read by login shell, after `.cshrc' at login. # see also csh(1), environ(7). So simply moving your commands from .cshrc to .login should be fully sufficient. -- Polytropon Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Any way to have login output AND use rsync/scp ?
On Sun, 1 May 2011 15:58:45 -0700 (PDT), George Sanders wrote: > I have my .cshrc file run some basic netstat and 'w' commands so that when I > log > in, I can see at a glance what is going on on the system and notice any > unusual > login activity, etc. > > However this completely breaks both scp and rsync - they cannot function at > all > (apparently) with any kind of stdio output from the shell. > > Is there any way around this ? Create a file ~/.login and put your commands (in sh syntax, not csh) there. This file will only be executed at interactive logins. See "man csh", section FILES for details. > Is there some way to specify an alternate login shell when I use scp or rsync > ? I'm sure the login shell has to be defined by the system, usually by the /etc/passwd file, field "shell"; therefore see "man 5 passwd". > Or conversely: > > Is there some way for my login shell (csh) to notice that it is scp or rsync > accessing the system, and skip the text output for them ? I don't think so, but it should work fine when you move the commands out of the shell configuration file (which is also read by script shells AND interactive shells) to the login startup file (which is only executed if the shell is an interactive login shell). -- Polytropon Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Any way to have login output AND use rsync/scp ?
> Date: Sun, 1 May 2011 15:58:45 -0700 (PDT) > From: George Sanders > Subject: Any way to have login output AND use rsync/scp ? > > I have my .cshrc file run some basic netstat and 'w' commands so that > when I log in, I can see at a glance what is going on on the system and > notice any unusual login activity, etc. > > However this completely breaks both scp and rsync - they cannot function > at all > (apparently) with any kind of stdio output from the shell. > > Is there any way around this ? > > Is there some way to specify an alternate login shell when I use scp or > rsync ? > > Or conversely: > > Is there some way for my login shell (csh) to notice that it is scp or > rsync accessing the system, and skip the text output for them ? > > Thanks. > ___ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions To > unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org" > ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Any way to have login output AND use rsync/scp ?
I have my .cshrc file run some basic netstat and 'w' commands so that when I log in, I can see at a glance what is going on on the system and notice any unusual login activity, etc. However this completely breaks both scp and rsync - they cannot function at all (apparently) with any kind of stdio output from the shell. Is there any way around this ? Is there some way to specify an alternate login shell when I use scp or rsync ? Or conversely: Is there some way for my login shell (csh) to notice that it is scp or rsync accessing the system, and skip the text output for them ? Thanks. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Sorry state of the rsync based CVS,replication
On Sun, 2010-11-14 at 12:13 +0100, Simon L. B. Nielsen wrote: > There is nothing which prevents mirror sites from providing access to > the CVS repo via rsync, even if they get it via CVSup... I went ahead with adding this to ftp2.freebsd.org: % rsync ftp2.freebsd.org::FreeBSD-CVS/ drwxr-xr-x 512 2010/08/02 13:35:40 . drwxr-xr-x 512 2010/08/02 22:09:36 gnats drwxr-xr-x 512 2010/08/03 02:34:06 mail drwxr-xr-x 512 2010/08/02 21:50:04 ncvs drwxr-xr-x 512 2010/08/03 00:51:26 www % -- Ken Smith - From there to here, from here to | kensm...@buffalo.edu there, funny things are everywhere. | - Theodor Geisel | signature.asc Description: This is a digitally signed message part
Re: Sorry state of the rsync based CVS,replication
On 12 Nov 2010, at 09:47, Patrick Bihan-Faou wrote: > Don't take this as flamebait, because I have no intention in starting a war > on this particular issue, but as good as cvsup is, this is unfortunately a > fairly isolated tool that, from my prospective (which is necessarily biaised > and incomplete), does not offer any feature compelling enough to prefer it > over rsync in our case. That position is by essence just a personal view, > applicable to me only and not to anybody else. Also I have to admit that now > that the m3 dependency is gone with csup, it becomes easier to return to it. The issue is not to remove CVS via rsync - just to remove it from the FTP collection where it doesn't belong. There is nothing which prevents mirror sites from providing access to the CVS repo via rsync, even if they get it via CVSup... If it's useful (IE, any of the primary mirrors requests it) we can probably rather easily set up rsync access via cvsup-master. That said, I think rsync access is likely not too interesting for most master mirrors as they likely provide access to the repo via CVSup already, so they have cvsup installed already. -- Simon L. B. Nielsen Hat: FreeBSD.org clusteradm ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Sorry state of the rsync based CVS,replication
Hi, [...] Regarding the permission of the Attic subdirs in place > The development/ section of the FTP site is something I hadn't looked at before so it took me a little time to find what populates it and investigate a little. I *think* the issue with the Attic directories not including world-read permissions was either an issue with a badly formed chmod(1) done a long time ago or an issue with the mechanism that populates that portion of the FTP site missing a umask setting in the script that does it some time back in history (it's there now). Not all of the Attic directories had the wrong permissions, it seemed to stop some time in 2007. I adjusted the permissions on ftp-master so hopefully this issue is fixed. However ... Great news. I'll check various rsync source later and see if the situation improves. We are moving to svn and svnsync for the freebsd source tree (and I am happy with this), but the ports do not seem to be available using SVN (or not in a documented way). Can something be done to restore RSYNC mirroring of the CVS tree to a working state ? The FTP site desperately needs to go on a diet so we're poking around to see if there is some stuff that can be dropped. This section of the site is a candidate for being removed. As you say the ports are not available in SVN but I'm curious why you use the content from the FTP site instead of just using a CVSUP mirror. Is there some benefit to it? We would sort of like to stop providing this as part of the FTP site if there really isn't any benefit to it over using the cvsup mirror infrastructure which won't be going away any time soon. The benefit is organisational to us. We did use cvsup in the past but it has been a pain to maintain as all the other external sources we keep in sync with use rsync. That combined with the requirement for m3 etc. for the sole purpose of syncing the CVS tree when there are alternatives (rsync for the freebsd cvs tree and more recently svnsync) made the switch to rsync a no brainer (note that this decision was taken long before csup came to life). Don't take this as flamebait, because I have no intention in starting a war on this particular issue, but as good as cvsup is, this is unfortunately a fairly isolated tool that, from my prospective (which is necessarily biaised and incomplete), does not offer any feature compelling enough to prefer it over rsync in our case. That position is by essence just a personal view, applicable to me only and not to anybody else. Also I have to admit that now that the m3 dependency is gone with csup, it becomes easier to return to it. Now if the plan is to eliminate rsync CVS mirroring (and I am in no position to criticize such a decision that is very well justified in your mail), we will of course adapt and go to one of the supported methods for CVS tree mirroring. If this means cvsup only, then we will do it without any hard feeling. I already figured that we must be almost the only ones to use rsync for CVS tree mirroring (otherwise that issue would have been detected and fixed a long time ago), and I don't expect (nor demand) the project to maintain a service for just one user, this would be ridiculous and unreasonable. Anyways, thank you very much for taking the time to look at this issue and hopefully having fixed it. This is really greatly appreciated. Best regards, Patrick. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Sorry state of the rsync based CVS,replication
On Wed, 2010-10-27 at 14:45 +0200, Patrick Bihan-Faou wrote: > We use FreeBSD extensively and keep a local mirror of the CVS > repository. Up until recently things where working properly with the > various servers listed in > http://www.freebsd.org/doc/handbook/mirrors-rsync.html, but sometime > during the summer ftp13.freebsd.org did not respond anymore and since > then rsync replication is broken. > > The main issue (besides the removal of ftp13.freebsd.org) is that most > rsync sources refuse to replicate the content of the .Attic directories > in the CVS tree. This means that performing a check-out on ports using a > tag usually won't work as some files will not be there anymore. > > Here are the typical logs I get using most rsync servers: > > rsync: opendir > "/3/freebsd-core/development/FreeBSD-CVS/ports/chinese/pcmanx/files/Attic" > (in vol) failed: Permission denied (13) > > At this moment the only rsync server that provides an adequate > replication of the CVS repository is ftp2.tw.FreeBSD.org. As others have reported this was caused by the permissions on the Attic directories not including world read permission. For sites where it was working it's actually an indication they're not following "best practices" for a mirror site. It's typically a bad idea to have the thing that allows access to the content of the mirror site running with the same credentials as what keeps the mirror site up to date. We don't use the 'feature' that allow for (pre-staging content that the world shouldn't have access to for a period of time, allowing the mirror sites to get fully populated before the release date) but I know of other projects that do. The ftp-master machines don't have that in place because they're not public and they need to allow the blessed mirrors access to everything (for the purposes of pre-staging, if we were actually using that feature...). The development/ section of the FTP site is something I hadn't looked at before so it took me a little time to find what populates it and investigate a little. I *think* the issue with the Attic directories not including world-read permissions was either an issue with a badly formed chmod(1) done a long time ago or an issue with the mechanism that populates that portion of the FTP site missing a umask setting in the script that does it some time back in history (it's there now). Not all of the Attic directories had the wrong permissions, it seemed to stop some time in 2007. I adjusted the permissions on ftp-master so hopefully this issue is fixed. However ... > We are moving to svn and svnsync for the freebsd source tree (and I am > happy with this), but the ports do not seem to be available using SVN > (or not in a documented way). > > Can something be done to restore RSYNC mirroring of the CVS tree to a > working state ? The FTP site desperately needs to go on a diet so we're poking around to see if there is some stuff that can be dropped. This section of the site is a candidate for being removed. As you say the ports are not available in SVN but I'm curious why you use the content from the FTP site instead of just using a CVSUP mirror. Is there some benefit to it? We would sort of like to stop providing this as part of the FTP site if there really isn't any benefit to it over using the cvsup mirror infrastructure which won't be going away any time soon. Thanks. -- Ken Smith - From there to here, from here to | kensm...@buffalo.edu there, funny things are everywhere. | - Theodor Geisel | signature.asc Description: This is a digitally signed message part
Sorry state of the rsync based CVS,replication
Hi, We use FreeBSD extensively and keep a local mirror of the CVS repository. Up until recently things where working properly with the various servers listed in http://www.freebsd.org/doc/handbook/mirrors-rsync.html, but sometime during the summer ftp13.freebsd.org did not respond anymore and since then rsync replication is broken. The main issue (besides the removal of ftp13.freebsd.org) is that most rsync sources refuse to replicate the content of the .Attic directories in the CVS tree. This means that performing a check-out on ports using a tag usually won't work as some files will not be there anymore. Here are the typical logs I get using most rsync servers: rsync: opendir "/3/freebsd-core/development/FreeBSD-CVS/ports/chinese/pcmanx/files/Attic" (in vol) failed: Permission denied (13) At this moment the only rsync server that provides an adequate replication of the CVS repository is ftp2.tw.FreeBSD.org. We are moving to svn and svnsync for the freebsd source tree (and I am happy with this), but the ports do not seem to be available using SVN (or not in a documented way). Can something be done to restore RSYNC mirroring of the CVS tree to a working state ? Best regards, Patrick Bihan-Faou ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync: failed to set permissions on "": Function not implemented (38)
On Wed, 27 Oct 2010, Noah wrote: No response on the rsync mail list so I am trying here. I am trying to figure out how I can get my rsync client and server to coexist better. I am running into the following error: rsync: failed to set permissions on "": Function not implemented (38) What is the target filesystem? That kind of error can happen when filesystems don't implement all the features rsync expects, like FAT. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
rsync: failed to set permissions on "": Function not implemented (38)
Hi there,
No response on the rsync mail list so I am trying here.
I am trying to figure out how I can get my rsync client and server to
coexist better. I am running into the following error:
rsync: failed to set permissions on "": Function not
implemented (38)
and the command line is
/usr/bin/rsync --recursive --compress --human-readable --progress
--update --perms -avz '/Applications' -e 'ssh -p 22 -o
ServerAliveInterval=10' --relative --delete root@:/
What can I do to alleviate these errors?
Cheers,
Noah
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: snapshot rsync dump/restore
n dhert wrote: I was told one could do this using rsync and by using a snapshot it would even be faster (?) Also try http://rdiff-backup.nongnu.org/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: snapshot rsync dump/restore
On Thu, Dec 17, 2009 at 09:03:50AM +0100, n dhert wrote: > I want to clone a FreeBSD system on another system. > Say, Mondaymorning I use the dump(8) to make dumpfiles of all filesystems > (dumpofroot.dmp, dumpofvar.dmp, ...tmp.dmp, ...usr.dmp, ...home.dmp ) on an > external USB disk. Dumping /tmp is usually not necessary. > The original system keeps running. > Then Wednesday I setup FreeBSD on the new system, and use restore(8) to put > the contents of my dumpfiles in the filesystems of that clone system. If the new system has different hardware, do not forget to adjust /boot/loader.conf, or build a custom kernel. And you should definitely edit /etc/rc.conf if even for the hostname. > Then I still need to have the changes occured during Mondaymorning till > Wednesdayafternoon on the original system, to be put in some way on the > clone system, so that these are not lost. > How exactly would one best proceed? I would recommend rsync. > Is rsync save regarding soft-links ? Yes, if you use the '-a' flag. Roland -- R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) pgpXk89WkOmYO.pgp Description: PGP signature
Re: snapshot rsync dump/restore
n dhert wrote: I want to clone a FreeBSD system on another system. Say, Mondaymorning I use the dump(8) to make dumpfiles of all filesystems (dumpofroot.dmp, dumpofvar.dmp, ...tmp.dmp, ...usr.dmp, ...home.dmp ) on an external USB disk. The original system keeps running. Then Wednesday I setup FreeBSD on the new system, and use restore(8) to put the contents of my dumpfiles in the filesystems of that clone system. Then I still need to have the changes occured during Mondaymorning till Wednesdayafternoon on the original system, to be put in some way on the clone system, so that these are not lost. I was told one could do this using rsync and by using a snapshot it would even be faster (?) Is rsync save regarding soft-links ? How exactly would one best proceed? Presumably you did a level 0 dump to make your initial copy? Did you happen to use the -u flag to dump? ie. update /etc/dumpdates? If so, then you can just do an incremental dump of everything that has changed since. (This is standaard dump(1) functionality). So long as the filesystems on the second machine haven't changed in the meantime, you should be able to just restore the incremental dump on top of the original full backup to get things pretty much in synch. (Although watch out for files that were deleted between the full and the incremental dumps). If you didn't update /etc/dumpdates, then you can manually edit /etc/dumpdates to achieve the same result. Failing that, yes, you can use rsync to synchronise the filesystem state. You don't have to snapshot the original system, but it will help to get you a consistent point-in-time copy of the original server. To create a snapshotted and mounted directory tree see the section on 'snapshot' in mount(8). If you have several filesystems to rsync to the other machine, you should snapshot all of them and mount them one over the other in the same relation as the original filesystems. Then you can rsynch the whole snapshotted directory tree to your new server in one command. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
snapshot rsync dump/restore
I want to clone a FreeBSD system on another system. Say, Mondaymorning I use the dump(8) to make dumpfiles of all filesystems (dumpofroot.dmp, dumpofvar.dmp, ...tmp.dmp, ...usr.dmp, ...home.dmp ) on an external USB disk. The original system keeps running. Then Wednesday I setup FreeBSD on the new system, and use restore(8) to put the contents of my dumpfiles in the filesystems of that clone system. Then I still need to have the changes occured during Mondaymorning till Wednesdayafternoon on the original system, to be put in some way on the clone system, so that these are not lost. I was told one could do this using rsync and by using a snapshot it would even be faster (?) Is rsync save regarding soft-links ? How exactly would one best proceed? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync include and exclude
Thanks. That took care of the problem. Jay On Oct 7, 2009, at 3:02 PM, krad wrote: 2009/10/7 Jay Hall Ladies and Gentlemen, I am using rsync to backup some information and I am having some problem with including and excluding directories. I want to include everything in the user's mail directory and everything in the user's documents directory. Everything else should be excluded. And, it is possible to have multiple users on a single computer. Following is what I am using for my include/exclude patterns. + /Users/*/Library/Mail/ + /Users/*/Documents/ - /Users/*/Documents/* - /Users/*/Library/* This gives me close to the desired result. However, there are some extra files included which I do not need to backup which are stored in /Users/username directory. If I add -/Users/*, no files are backed up. After reading the man pages and several examples, I thought I had a handle on this. Any suggestions would be greatly appreciated. Thanks, Jay ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org " try adding this line to the top + /Users/* then this to the bottom - /Users/*/* I use lists like this at work a lot and you have to be careful. The ordering of the rules is very important. The common on for people to get wrong is the mysql rules.. We generally dont copy the binary db files as they would never be consistent, so we dump the db else where and exclude the files. Except we need to capture the my.cnf file. The following ruleset does it + /var/db/mysql + /var/db/mysql/my.cnf - /var/db/mysql/** If you deviate from this things dont tend to work. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync include and exclude
2009/10/7 Jay Hall > Ladies and Gentlemen, > > I am using rsync to backup some information and I am having some problem > with including and excluding directories. > > I want to include everything in the user's mail directory and everything in > the user's documents directory. Everything else should be excluded. And, > it is possible to have multiple users on a single computer. > > Following is what I am using for my include/exclude patterns. > > + /Users/*/Library/Mail/ > + /Users/*/Documents/ > - /Users/*/Documents/* > - /Users/*/Library/* > > This gives me close to the desired result. However, there are some extra > files included which I do not need to backup which are stored in > /Users/username directory. > > If I add -/Users/*, no files are backed up. > > After reading the man pages and several examples, I thought I had a handle > on this. Any suggestions would be greatly appreciated. > > Thanks, > > > Jay > > > ___ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscr...@freebsd.org" > try adding this line to the top + /Users/* then this to the bottom - /Users/*/* I use lists like this at work a lot and you have to be careful. The ordering of the rules is very important. The common on for people to get wrong is the mysql rules. We generally dont copy the binary db files as they would never be consistent, so we dump the db else where and exclude the files. Except we need to capture the my.cnf file. The following ruleset does it + /var/db/mysql + /var/db/mysql/my.cnf - /var/db/mysql/** If you deviate from this things dont tend to work. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
rsync include and exclude
Ladies and Gentlemen, I am using rsync to backup some information and I am having some problem with including and excluding directories. I want to include everything in the user's mail directory and everything in the user's documents directory. Everything else should be excluded. And, it is possible to have multiple users on a single computer. Following is what I am using for my include/exclude patterns. + /Users/*/Library/Mail/ + /Users/*/Documents/ - /Users/*/Documents/* - /Users/*/Library/* This gives me close to the desired result. However, there are some extra files included which I do not need to backup which are stored in /Users/username directory. If I add -/Users/*, no files are backed up. After reading the man pages and several examples, I thought I had a handle on this. Any suggestions would be greatly appreciated. Thanks, Jay ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: how to do a live migration of a freebsd box to another box with rsync
Hi, Thanks guys, everything worked perfectly ! - For the liveCD, i booted the second box with FreeNAS ( http://www.freenas.org/index.php?lang=fr ) , which include rsync and ssh :-) - Created the partition layout following the official doc http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/disks-adding.html. @Ruben: Thanks for your help btw for restoring the bootloader :-) - Then just rsynced the "/" filesystem excluding the /dev directory. - Ajusted /etc/fstab - Voilà ! Seems easier than a migration of GNU/Linux after all :) Thanks again for your help ! Cheers, ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: how to do a live migration of a freebsd box to another box with rsync
> Hi, > I'm used to migrate GNU/Linux system from one box to another by booting the > second box with a liveCD (like systemrescueCD for example) and by copying > the "/" filesystem (using the ssh transport) with rsync. > I would like to do the same for BSD system but i have two issues: > - as the UFS write support is still experimental in the Linux kernel, it > seems that i've to use a BSD liveCD but i can't find one :-/ I heard about > frenzy ( http://frenzy.org.ua/en/ ) but the homepage says that the project > is no longer maintained ! > - i'm wondering how to restore the bootloader after copying the files on the > second box. On linux, i can use the grub-install script to do the job but > i'm a bit lost on FreeBSD :-) Hello. The Frenzy distro is still quite usable albeit it was abandoned. ;) Also you can find official FreeBSD liveCD iso here: ftp://ftp.freebsd.org/pub/FreeBSD/releases/i.386/ISO-IMAGES/7.2/7.2-RELEASE-i386-livefs.iso (change arch type according to you platform). -- Best regards, Jeff | "Nobody wants to say how this works. | | Maybe nobody knows ..." | | Xorg.conf(5)| ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: how to do a live migration of a freebsd box to another box with rsync
On Sun, Jul 05, 2009 at 06:18:03PM +0200, insrc typed: > Hi, > I'm used to migrate GNU/Linux system from one box to another by booting the > second box with a liveCD (like systemrescueCD for example) and by copying > the "/" filesystem (using the ssh transport) with rsync. > I would like to do the same for BSD system but i have two issues: > - as the UFS write support is still experimental in the Linux kernel, it > seems that i've to use a BSD liveCD but i can't find one :-/ I heard about > frenzy ( http://frenzy.org.ua/en/ ) but the homepage says that the project > is no longer maintained ! > - i'm wondering how to restore the bootloader after copying the files on the > second box. On linux, i can use the grub-install script to do the job but > i'm a bit lost on FreeBSD :-) Assuming you install on the first slice of the first disk (ad0s1), to install the bootloader and bootstrap code: fdisk -B ad0 bsdlabel -B ad0s1 Ruben ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: how to do a live migration of a freebsd box to another box with rsync
Hi, Am Sonntag, 05. Jul 2009, 18:18:03 +0200 schrieb insrc: > - as the UFS write support is still experimental in the Linux kernel, it > seems that i've to use a BSD liveCD but i can't find one :-/ I heard about > frenzy ( http://frenzy.org.ua/en/ ) but the homepage says that the project > is no longer maintained ! There is a livefs with the original ISO images: ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/ISO-IMAGES/7.2/ I further found DesktopBSD but I didn't try that. I strongly recommend that you build yourself an USB stick. Here's what you need to do: http://typo.submonkey.net/articles/2006/4/13/installing-freebsd-on-usb-stick-episode-2 I went forth, chrooted into the stick and installed Vim, some diagnose/repair tools and an XFCE. I even managed to install Grub and let the user switch the boot process back to the hard disk. Further, I made a second partition named "transfer" formatted with FAT so that I can write some data from a Windows to it. I look enviously at the Grml project and I find it a great pity that there is no BSD equivalent. Bertram -- Bertram Scharpf Stuttgart, Deutschland/Germany http://www.bertram-scharpf.de ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: how to do a live migration of a freebsd box to another box with rsync
insrc wrote: > it seems that i've to use a BSD liveCD but i can't find one :-/ www.freesbie.org The site is not responding for me ATM, but the text is cached here: http://74.125.155.132/search?q=cache:WjK0Anp5tb4J:www.freesbie.org/+freesbie+freebsd&hl=en&gl=us&strip=1 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
how to do a live migration of a freebsd box to another box with rsync
Hi, I'm used to migrate GNU/Linux system from one box to another by booting the second box with a liveCD (like systemrescueCD for example) and by copying the "/" filesystem (using the ssh transport) with rsync. I would like to do the same for BSD system but i have two issues: - as the UFS write support is still experimental in the Linux kernel, it seems that i've to use a BSD liveCD but i can't find one :-/ I heard about frenzy ( http://frenzy.org.ua/en/ ) but the homepage says that the project is no longer maintained ! - i'm wondering how to restore the bootloader after copying the files on the second box. On linux, i can use the grub-install script to do the job but i'm a bit lost on FreeBSD :-) Thanks for your help ! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Fwd: rsync colon in filename to MSWin fails
Dagnabbit! Meant to also send this to the list. Kurt On Fri, Sep 19, 2008 at 10:28, Giorgos Keramidas wrote: > On Fri, 19 Sep 2008 13:37:50 +0100, Anton Shterenlikht > wrote: >> I cannot trasfer a file with a colon via rsync to a Win box. >> I've rsync-3.0.4 on the FBSD (sending side) and rsync-2.6.9 >> under cygwin on Win (receiving side). I'm not sure what the >> error message means: >> >> % rsync ./http:__en.wikipedia.org_favicon.ico me...@xxx.xxx.xxx.xxx: >> rsync: rename "/cygdrive/c/Documents and Settings/mexas/My >> Documents/work/.http:__en.wikipedia.org_favicon.ico.8dBX2K" -> >> "http:__en.wikipedia.org_favicon.ico": No such file or directory (2) >> rsync error: some files/attrs were not transferred (see previous errors) >> (code 23) at main.c(1040) [sender=3.0.4] >> >> Note the extra dot before the file name, and extra suffix at the >> end. Is this the expected behaviour? > > Yes, this is the expected broken behavior of Windows. The ':' character > cannot be used in a normal filename in Windows. The bug goes way back, > when DOS developers chose to use ':' to denote 'drive names'. > >> However, to another FBSD box transfer is fine: >> >> % rsync ./http:__en.wikipedia.org_favicon.ico me...@zzz.zzz.zzz.zzz: >> % > > UNIX only reserves '/' as the path separator (and in some cases the '\0' > character for the end-of-fiename string marker). So the ':' character > is valid for file names or directory names. > >> Is this something to do with Windows, or old rsync version on cygwin? > > Yes, it has to do with Windows. It does indeed have to do with Windows - see this article: http://msdn.microsoft.com/en-us/library/aa365247.aspx However, there's another thing to know about the use of colons in file names, and it's not simply to do with driver letters - NTFS has a facility called Alternate Data Streams, which was implemented to support Macintoshes, originally. See this article for a start: http://msdn.microsoft.com/en-us/library/aa364404(VS.85).aspx ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync colon in filename to MSWin fails
Here's a good workaround. This works for me from Linux to Win32. In your rsync script, before the rsync, do either find ./ -name '*:*' | xargs tar -czvf colon_files.tgz or (if you're doing the whole box as an rsync backup, like I am) do updatedb locate : | xargs tar -czvf colon_files.tgz Make sure that colon_files.tgz is in the path that's getting backed up by rsync. Then you can do the --exclude '*:*' in your rsync job to skip the files you just tarred. I tested this with the locate version above and it works. Also, updatedb & locate should go much faster than find if you want to do the whole box. I'm not sure if there's a way to use locate to just search a subdirectory, though. Best, Dave Anton Shterenlikht wrote: > > I cannot trasfer a file with a colon via rsync to a Win box. > I've rsync-3.0.4 on the FBSD (sending side) and rsync-2.6.9 > under cygwin on Win (receiving side). I'm not sure what the > error message means: > > % rsync ./http:__en.wikipedia.org_favicon.ico me...@xxx.xxx.xxx.xxx: > rsync: rename "/cygdrive/c/Documents and Settings/mexas/My > Documents/work/.http:__en.wikipedia.org_favicon.ico.8dBX2K" -> > "http:__en.wikipedia.org_favicon.ico": No such file or directory (2) > rsync error: some files/attrs were not transferred (see previous errors) > (code 23) at main.c(1040) [sender=3.0.4] > > Note the extra dot before the file name, and extra suffix at the > end. Is this the expected behaviour? > > However, to another FBSD box transfer is fine: > > % rsync ./http:__en.wikipedia.org_favicon.ico me...@zzz.zzz.zzz.zzz: > % > > Is this something to do with Windows, or old rsync version on cygwin? > > Even more bizzare, I get the same error even if I "--exclude" all > such files from transfer. > > many thanks > anton > > -- > Anton Shterenlikht > Room 2.6, Queen's Building > Mech Eng Dept > Bristol University > University Walk, Bristol BS8 1TR, UK > Tel: +44 (0)117 928 8233 > Fax: +44 (0)117 929 4423 > ___ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscr...@freebsd.org" > > -- View this message in context: http://www.nabble.com/rsync-colon-in-filename-to-MSWin-fails-tp19571614p23925088.html Sent from the freebsd-questions mailing list archive at Nabble.com. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync approach
>> On Wed, 27 May 2009 15:03:30 -0700,
>> prad said:
P> We are thinking of rsync to duplicate 1st [box] > 2nd [box] (with the
P> exception of rc.conf and a few other files of course because we don't
P> want them to be absolutely identical).
P> we plan to allow root login and have disabled all password access so
P> that rsync can preserve permissions. is this a good way to accomplish
P> the bkp job?
If you're going to use root login, I'd suggest access control for ssh
via either daemontools or tcpwrappers, and add some extra security
by putting 'from="hostname"' in root's entry in the "authorized_keys2"
file:
from="1st.box.com" ssh-dss B3NzaC1MtH[...]WDXDrq03pE= r...@1st.box.com
It's not strictly necessary to allow root connections if you want
to keep permissions intact. I use an unprivileged account ("bkup")
to copy gzipped cpio archives between systems. On the 1st box, root
can use pax or cpio to create the archive, and then run something as
user "bkup" to do the copy to the 2nd box:
root# cd /some/where
root# find . -print | pax -x cpio -wd | gzip -1c > /tmp/arch.pax.gz
root# su bkup -c "scp -c arcfour -i /bkup/.ssh/backuphost_dsa \
/tmp/arch.pax.gz 2nd.box.com:/someplace/bkup/can/write"
The arcfour cipher will probably give you better throughput. To unpack
the files on 2nd.box.com:
root# cd /some/where/else
root# gunzip -c /someplace/bkup/can/write/arch.pax.gz | pax -rd -pe
root# rm /someplace/bkup/can/write/arch.pax.gz
If the files you're syncing are huge, you're better off using root login
plus rsync.
--
Karl Vogel I don't speak for the USAF or my company
SUVs are gross because they're the solution to a gross problem:
how to make minivans look more masculine. --Paul Graham
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync approach
we have 2 static ip addresses with a machine running 7.2 connected to each. one is the primary server, while the other does only dns and receives bkp dumps from the first. we want to set things up so the 2nd can be brought on line at a moment's notice. therefore, we are thinking of rsync to duplicate 1st > 2nd (with the exception of rc.conf and a few other files of course because we don't want them to be absolutely identical). we plan to allow root login and have disabled all password access so that rsync can preserve permissions. i don't catch why disabling password access will allow rsync to preserve permission. It will preserve just when you give proper option is this a good way to accomplish the bkp job? yes it is. There is another way too - having both adventage and disadventage. 1) make an option in FreeBSD loader menu to run ramdisk-freebsd (ramdisk from file). Put on that cutdown ramdisk system only startup of ggated with a disk 2) on main machine run ggatec and gmirror. you will get network mirrored hard disk. make this procedure conditional so it runs only on first machine (for eg check MAC address of your network interface) in case of machine 1 fail, you just run second with normal, instead of ramdisk mode. It has adventage of full realtime replication, but it's disadventage in the same time. For example if you run rsync once per 2 hours, and you by accident delete a lots of things, you can recover. with gmirror way it is instantly replicated so you can't recover ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: rsync approach
On 27 May 2009, at 15:03, prad wrote: we have 2 static ip addresses with a machine running 7.2 connected to each. one is the primary server, while the other does only dns and receives bkp dumps from the first. we want to set things up so the 2nd can be brought on line at a moment's notice. therefore, we are thinking of rsync to duplicate 1st > 2nd (with the exception of rc.conf and a few other files of course because we don't want them to be absolutely identical). we plan to allow root login and have disabled all password access so that rsync can preserve permissions. is this a good way to accomplish the bkp job? It might be, but its difficult to say without knowing a lot more about what going on in the system. For example, rsync of large files that change real often can be an issue. You may get an inconsistent copy of the file if it is changing during the update of that file. Rsync does one file at a time. Hence if you have multiple files that need to be consistent between them it might not happen. If transactions changing a file occur between the rsync of the first file and the rsync of the second file, they will not be consistent. I use rsync for basically the same thing you are considering. However, in my situation, the real dynamic files are database files. I don't rsync them. Those are exported every evening and those files are rsync'd. During the day, every application that updates a database file also adds a copy of the new record to the end of a log file. That log file is also rsync'd about every 5 minutes. In this way I can recover to within 5 minutes quite easily. The last 5 minutes might be a bit more work, but the information would be available from the original sources at that point. Allowing root login is generally not a great approach. Rsync will retain permissions (use -p) if the user id's and group ids are the same on both systems. You may have to modify some of the pam files to permit rsync to function easily. The easy test is to rsh date If that works and you get the date, then rsync will function properly. If not you need to track down why in the various log files. I had to make the following change: pam.d/rsh: -auth requiredpam_rhosts.so no_warn +auth required pam_rhosts.so no_warnallow_root This was first done quite a few years ago and has been propagated to the current systems so I don't know if its still required or not. Our systems only have 3 users (the administrators) so we have passwords working fine. It has not interfered with rsync. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
rsync approach
we have 2 static ip addresses with a machine running 7.2 connected to each. one is the primary server, while the other does only dns and receives bkp dumps from the first. we want to set things up so the 2nd can be brought on line at a moment's notice. therefore, we are thinking of rsync to duplicate 1st > 2nd (with the exception of rc.conf and a few other files of course because we don't want them to be absolutely identical). we plan to allow root login and have disabled all password access so that rsync can preserve permissions. is this a good way to accomplish the bkp job? -- In friendship, prad ... with you on your journey Towards Freedom http://www.towardsfreedom.com (website) Information, Inspiration, Imagination - truly a site for soaring I's ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Using rsync for versioned backups without --backup
On Sun, May 24, 2009 at 11:39:57PM -0700, Kelly Jones wrote: > I want to use rsync to backup a large file (say 1G) that changes a > little each day (say 1M), but I also want the ability to re-create > older versions of this file. > > I could use --backup, but that would create a 1G file each day, even > though I only "really" need the 1M that's changed. > > How do I tell rsync: "while updating, also store the changes you'd > need to convert today's backup into yesterday's backup"? I don't think rsync can do that. Essentially it is a file copying tool. You could use diff if it is a text-only file, or xdelta if it is a binary file. But both would require you to keep at least two subsequent versions of the file so a diff can be generated. You'd need to do something like this every day: diff -u foo-yesterday foo >diff-20090525 # save the diff somewhere rm foo-yesterday cp foo foo-yesterday Another possibility is to control the file with a revision control system. If the file is plain text, rcs(1) will work. If it is a binary file use a system like devel/git that handles binary files well. Say that your file is called . Since git tracks directory contents, best put it in a separate directory, and put that under git control: mkdir ~/foo mv bar ~/foo/ cd ~/foo git init git add bar git commit -a -m "Initial commit" So now you can start changing the file. Next day you see if it has changed, and if so, check in the changes: cd ~/foo git status git commit -m "Changes 2009-05-25" bar If you now make a backup of ~/foo/.git, you can always restore every checkin you did of . Roland -- R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) pgpZ72158oy5H.pgp Description: PGP signature
Using rsync for versioned backups without --backup
I want to use rsync to backup a large file (say 1G) that changes a little each day (say 1M), but I also want the ability to re-create older versions of this file. I could use --backup, but that would create a 1G file each day, even though I only "really" need the 1M that's changed. How do I tell rsync: "while updating, also store the changes you'd need to convert today's backup into yesterday's backup"? I realize I could use diff or something, but since rsync has to calculate minimal changes anyway, it'd be nice to store them. I thought the --itemize-changes option might do this, but no. -- We're just a Bunch Of Regular Guys, a collective group that's trying to understand and assimilate technology. We feel that resistance to new ideas and technology is unwise and ultimately futile. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Rsync always preserve owner/group
Hi list! I'm in the process of creating an automated update system based on SVN and rsync. I 'svn update' every night and then rsync the updated files from the working copy of SVN to the right folders. The problem I have is that when I rsync the files, the destination owner/group is set with the owner/group from the source file (the file from the SVN with a wrong owner/group). I don't want to modify the destination file owner/group but only update it's content. I'm not using any option with the rsync command. I also observed that the permissions are not changed in the transfer, which is a normal behaviour and perfect in my situation. Example: ll /usr/local/etc/scripts/ -rwxrwxrwx 1 psm psm2754 Mar 17 10:51 wakeup.sh And the details of the source file: ll /root/update/scripts/ -rwxr-xr-x 1 root wheel 2754 Mar 13 11:37 wakeup.sh Execution of the rsync command: rsync /root/update/scripts/wakeup.sh /usr/local/etc/scripts/ ll /usr/local/etc/scripts/ -rwxrwxrwx 1 root wheel 2754 Mar 17 10:59 wakeup.sh I also tried to explicitly disable preserve group with --no-g but got the same result. What am I doing wrong? Thanks a lot for your help! Martin ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Rsync | Push script
> > On Sat, Feb 21, 2009 at 02:40:21PM +0100, Jos Chrispijn wrote: > > > > Just having made a backup script that should take care of nocturnal > > backup of my mySQL data from one server to my backup server. > > > > cd /backup > > DATE=`date "+%d%m%y"` > > DIR=backup.$DATE > > > > /letc/rc.d/mysql-server stop > > > rsync -avpog /var/db/mysql//r...@10.10.10.50:123/usr/backup/$DATE/ > > /letc/rc.d/mysql-server start > > > > > It goes wrong when I run the rsync line; I run my backup thru port 123 > > (can be any portnumber). > > 10.10.10.50 is backup server on which I want to logon as root; during > > script run I will fill out root password myself. > > > > Can someone hint me in the right direction? rsync deamon is running. > > > > Jos Chrispijn > Doesn't rsync work via ssh by default? The default configuration of sshd on the backup server probably prevents external root users from accessing the server. Are you sure you want to run the backups as root? Andrew Gould ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Rsync | Push script
On Sat, Feb 21, 2009 at 02:40:21PM +0100, Jos Chrispijn wrote: > > Just having made a backup script that should take care of nocturnal > backup of my mySQL data from one server to my backup server. > > cd /backup > DATE=`date "+%d%m%y"` > DIR=backup.$DATE > > /letc/rc.d/mysql-server stop > rsync -avpog /var/db/mysql//r...@10.10.10.50:123/usr/backup/$DATE/ > /letc/rc.d/mysql-server start ^^^^ > > It goes wrong when I run the rsync line; I run my backup thru port 123 > (can be any portnumber). > 10.10.10.50 is backup server on which I want to logon as root; during > script run I will fill out root password myself. > > Can someone hint me in the right direction? rsync deamon is running. > > Jos Chrispijn Your script looks like it's got some typos in it (marked above). Don't think that has anything to do with your problems by the sounds of it. Regards, -- Frank Contact info: http://www.shute.org.uk/misc/contact.html ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Rsync | Push script
Just a small note which has nothing to do with the actual rsync problem: On Sat, 21 Feb 2009 14:40:21 +0100, Jos Chrispijn wrote: > DATE=`date "+%d%m%y"` In order to be able to sourt your backups by date, you could use the form DATE=`date "+%y%m%d"` or DATE=`date "+%Y%m%d"` to get a date signature that can be sorted. Just a suggestion - I've had the best experiences with the form DATE=`date "+%Y-%m-%d"` attached to the backup's "subject". -- Polytropon >From Magdeburg, Germany Happy FreeBSD user since 4.0 Andra moi ennepe, Mousa, ... ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Rsync | Push script
On Sat, Feb 21, 2009 at 06:14:27PM +0100, Jos Chrispijn wrote: > Sorry for the confusion; the problem is in the rsync line: > > rsync -avpog /var/db/mysql //r...@10.10.10.50:123/usr/backup/ > > Running this line causes rsync to say: > Unexpected remote arg: r...@10.10.10.50:123 > rsync error: syntax or usage error (code 1) at main.c(1202) [sender=3.0.5] > > On the server that I use this push script, I want to connect as root > to the backup server; thru port 123 (can be any port). With rsync > running I can't get this to work. Can someone tell me what I do wrong > here? You got the syntax for the rsync command wrong. Either remove the '//' in front of the remote address or use 'rsync://'. See the rsync(1) manual page. Roland -- R.F.Smith http://www.xs4all.nl/~rsmith/ [plain text _non-HTML_ PGP/GnuPG encrypted/signed email much appreciated] pgp: 1A2B 477F 9970 BA3C 2914 B7CE 1277 EFB0 C321 A725 (KeyID: C321A725) pgpL5uR37bpx6.pgp Description: PGP signature
Re: Rsync | Push script
Sorry for the confusion; the problem is in the rsync line: rsync -avpog /var/db/mysql //r...@10.10.10.50:123/usr/backup/ Running this line causes rsync to say: Unexpected remote arg: r...@10.10.10.50:123 rsync error: syntax or usage error (code 1) at main.c(1202) [sender=3.0.5] On the server that I use this push script, I want to connect as root to the backup server; thru port 123 (can be any port). With rsync running I can't get this to work. Can someone tell me what I do wrong here? Jos Chrispijn ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: Rsync | Push script
Jos Chrispijn wrote: > > Just having made a backup script that should take care of nocturnal > backup of my mySQL data from one server to my backup server. > > cd /backup > DATE=`date "+%d%m%y"` > DIR=backup.$DATE > > /letc/rc.d/mysql-server stop > rsync -avpog /var/db/mysql//r...@10.10.10.50:123/usr/backup/$DATE/ > /letc/rc.d/mysql-server start > > It goes wrong when I run the rsync line; I run my backup thru port 123 > (can be any portnumber). > 10.10.10.50 is backup server on which I want to logon as root; during > script run I will fill out root password myself. > > Can someone hint me in the right direction? rsync deamon is running. Based on your description, I have no idea what your actual problem is. I mean, you're basically saying, "it doesn't work." Which doesn't give folks any way to help you, really. What error message do you see? What's in the logs? You're running the rsync client with -v, which should be pretty verbose, have you tried running the rsync server with the logging amped up? -- Bill Moran http://www.potentialtech.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
