DO NOT REPLY [Bug 4793] link-dest hardlink does not always work well with -o -g -p
https://bugzilla.samba.org/show_bug.cgi?id=4793 --- Comment #3 from [EMAIL PROTECTED] 2007-07-20 00:25 CST --- > You can accomplish this yourself by filling the destination with hard-links > from the previous backup in advance using "cp -al". I'll give this a shot. > You might also consider representing the source attributes in a separate file > in the backup so that there is no need to use -ogp. Is there a tool to save permissions and then reapply them? This would work for me. > I think it's odd to want to preserve attributes in the current backup but > not care about attributes in previous backups being overwritten. I honestly didn't think it that odd. To explain: When we need to restore an entire file system due to hardware error, it will without exception be from the last available backup. In this case it is desirable to restore all permissions. The main/only reason to go to older snapshots is if a user needs an older copy of the data within one of his files where the latest backup is insufficient. In this case we could restore the backup file over top of his current file if it exists while keeping the current permissions (this might be preferred even if backup permissions were available). > I don't see a strong argument for building support for this into rsync. Fair enough. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 4793] link-dest hardlink does not always work well with -o -g -p
https://bugzilla.samba.org/show_bug.cgi?id=4793 --- Comment #2 from [EMAIL PROTECTED] 2007-07-20 00:22 CST --- > You can accomplish this yourself by filling the destination with hard-links > from the previous backup in advance using "cp -al". I'll give this a shot. > You might also consider representing the source attributes in a separate file > in the backup so that there is no need to use -ogp. Is there a tool to save permissions and then reapply them? This would work for me. > I think it's odd to want to preserve attributes in the current backup but > not care about attributes in previous backups being overwritten. I honestly didn't think it that odd. To explain: When we need to restore an entire file system due to hardware error, it will without exception be from the last available backup. In this case it is desirable to restore all permissions. The main/only reason to go to older snapshots is if a user needs an older copy of the data within one of his files where the latest backup is insufficient. In this case we could restore the backup file over top of his current file if it exists while keeping the current permissions (this might be preferred even if backup permissions were available). > I don't see a strong argument for building support for this into rsync. Fair enough. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RSYNC Hung
Hi all, RSYNC Server: CENT OS 5 RSYNC Version: 2.6.8, Running RSYNC Daemon. Client: Windows 2003 Standard, Delta Copy running rsync.exe version 2.6.6 We are backing up 2 files at each site at a total of 3 sites. 1 site is running perfectly, whilst the other 2 are causing constant grief. The backups at these 2 sites are hanging during the RSYNC process at random places. One of the two files cause the hang. We are running the following command on the client to initiate the backup: rsync.exe -v -rlt -z --delete "/cygdrive/D/MT32/ddiBkRes/Backup/PHCCAMMT32.GBK" "192.168.X.X::rsync//PHCCAMMT32.GBK" When it hangs, If I kill the rsync process on the client it will attempt the backup again. Usually a few retries and it will go through successfully. The RSYNC client log just states "Syntax or usage errorRsync.exe returned an error. Will try again. This is retry number 1 of 5" On the server, I have ran strace on the 2 rsync daemon processes and getting the following output after killing the rsync process on the client. [EMAIL PROTECTED] ~]# strace -c -T -p 4088 Process 4088 attached - interrupt to quit Process 4088 detached % time seconds usecs/call callserrors syscall -- --- --- - - 68.700.004287 17924 select 16.270.001015 851212 open 6.440.000402 101 4 send 4.360.000272 54 5 read 3.480.000217 54 4 time 0.750.47 24 2 1 waitpid 0.000.00 0 1 1 kill 0.000.00 0 1 sigreturn 0.000.00 0 2 rt_sigaction -- --- --- - - 100.000.0062405514 total [EMAIL PROTECTED] ~]# [EMAIL PROTECTED] ~]# strace -c -T -p 4089 Process 4089 attached - interrupt to quit Process 4089 detached % time seconds usecs/call callserrors syscall -- --- --- - - 58.36 15.742245 690 22815 1 read 40.37 10.888016 871 12501 write 1.250.337640 217 1558 select 0.020.005026 7765 _llseek 0.000.000275 138 2 rt_sigaction -- --- --- - - 100.00 26.973202 36941 1 total [EMAIL PROTECTED] ~]# [EMAIL PROTECTED] ~]# These logs don't make much sense to me as my Linux skills are limited. For further troubleshooting I was thinking of trying the backups without compression and using CYGWIN in conjunction with the newest version of RSYNC. Any Help would be great. Troy -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rysnc Schedule
About 4 hours after my reply I had a functional rsnapshot install - what took me so long (besides having 'real' work to do ;) was trying to figure out how to split up my vpopmail/domains directories so the rsync wouldn't take so long (because of so many files/directories). I ended up using the 'include_conf' directive, and dynamically creating a file with just backup points. It seems like a great solution, and was easy to install and setup. Thanks! Rick On Jul 19, 2007, at 6:09 PM, sparty2809 wrote: Sorry, I didn't mean to come off rude. I appreciate the help. I will look into it and see how it works. Thanks for all the suggestions. Matt McCutchen-3 wrote: On 7/19/07, Chris <[EMAIL PROTECTED]> wrote: I know how to use the command prompt, was just curious if it had a gui. OK. Please don't take any offense at my offering a little extra support in case you were a user who needed it. So this can backup for 30 days and once 30 days comes up, it will delete the 1st day and add the 31st? Yes. Just set a backup interval of "daily" with 30 snapshots in the configuration file. If you like, you can also add other intervals, e.g., weekly snapshots for the past 10 weeks. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart- questions.html -- View this message in context: http://www.nabble.com/Rysnc-Schedule- tf4104577.html#a11699492 Sent from the Samba - rsync mailing list archive at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/ listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart- questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rysnc Schedule
Sorry, I didn't mean to come off rude. I appreciate the help. I will look into it and see how it works. Thanks for all the suggestions. Matt McCutchen-3 wrote: > > On 7/19/07, Chris <[EMAIL PROTECTED]> wrote: >> I know how to use the command prompt, was just curious if it had a gui. > > OK. Please don't take any offense at my offering a little extra > support in case you were a user who needed it. > >> So this can backup for 30 days and once 30 days comes up, it will delete >> the >> 1st day and add the 31st? > > Yes. Just set a backup interval of "daily" with 30 snapshots in the > configuration file. If you like, you can also add other intervals, > e.g., weekly snapshots for the past 10 weeks. > > Matt > -- > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > > -- View this message in context: http://www.nabble.com/Rysnc-Schedule-tf4104577.html#a11699492 Sent from the Samba - rsync mailing list archive at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Compression error?
On Thu, 19 Jul 2007, WCSL wrote: > Seems to work fine without compression -z flag omitted. Is this going through a Watchguard firewall at any point? ..Chuck.. -- http://www.quantumlinux.com Quantum Linux Laboratories, LLC. ACCELERATING Business with Open Technology "The measure of the restoration lies in the extent to which we apply social values more noble than mere monetary profit." - FDR 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync bug?? (rsync fails when -C is used).
On 7/19/07, Paul Cui <[EMAIL PROTECTED]> wrote: hi, All. I just compiled the latest rsync (2.6.9). but I'm getting an error when I use the -C option. eg: $ rsync -aCv host1:/home/john/data/ /home/john/data receiving file list ... ERROR: out of memory in add_rule [sender] rsync error: error allocating core memory buffers (code 22) at util.c(115) [sender=2.6.9] rsync: connection unexpectedly closed (8 bytes received so far) [receiver] rsync error: error in rsync protocol data stream (code 12) at io.c(453) [receiver=2.6.9] The remote rsync is running out of memory as it collects the CVS ignore rules (add_rule). Something could be awry with the CVS ignore rules, or the machine could just be low on memory. Please run rsync again with -vvv (three verbose options), which will make the remote rsync show (among other useful information) the individual calls to add_rule, and send the resulting output to the list. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Compression error?
On 7/19/07, WCSL <[EMAIL PROTECTED]> wrote: Error received when trying to use rsync -vz on the client inflate (token) returned -5 rsync error: error in rsync protocol data stream (code 12) at token.c(458) rsync: writefd_unbuffered failed to write 4092 bytes: phase "unknown" [generator]: Broken pipe (32) rsync error: error in rsync protocol data stream (code 12) at io.c(909) Seems to work fine without compression -z flag omitted. There have been occasional reports of -z getting corrupted in the past, but we've never gotten a reproducible case to investigate and find the underlying problem. If you run the command repeatedly, does that same error occur every time? Even better if it still happens with the same files on a different pair of machines. If so, please send the source and destination files prior to the run to Wayne Davison <[EMAIL PROTECTED]>, the rsync maintainer, unless they're big and/or private in which case you should discuss other arrangements with him. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync bug?? (rsync fails when -C is used).
hi, All.
I just compiled the latest rsync (2.6.9). but I'm getting an error when
I use the -C option.
eg:
$ rsync -aCv host1:/home/john/data/ /home/john/data
receiving file list ... ERROR: out of memory in add_rule [sender]
rsync error: error allocating core memory buffers (code 22) at util.c(115)
[sender=2.6.9]
rsync: connection unexpectedly closed (8 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(453)
[receiver=2.6.9]
and if I remove the -C option. everything is fine. the directory structure is
very simple with no sym/hard links. and total number of files in there is about
30. running the command on local file system has no problem (replicating local
dir with -C option).
I did truss the remote sshd and see the client connected successfully. forked
child sshd which forked again and rsync exec'd:
137:execve("/usr/bin/rsync", 0x0004509C, 0x000450F8) argc = 6
137: argv: rsync --server --sender -vlogDtprC . /home/john/data
137: envp: _=/usr/bin/rsync
Both end of rsync is version 2.6.9. and both system are running solaris 9.
binary compiled with Sun's cc (not gcc).
The thing that makes me think that this is a bug in rsync is that I also tried
rsync to another solaris 9 machine which is running rsync 2.5.0. and it works
with no problem (with the -C option).
I goggled around.. and most people who are getting this error was dealing with
large number of file/directory.
Any info on this would be helpful.
thanks.
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Compression error?
Hello, Server rsyncd [EMAIL PROTECTED] steve]# rsync --version rsync version 2.6.9 protocol version 29 [EMAIL PROTECTED] steve]# uname -a Linux data1.contact-24-7.local 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 26 14:30:58 EDT 2007 i686 i686 i386 GNU/Linux [EMAIL PROTECTED] steve]# Client rsync [EMAIL PROTECTED] cron.daily]# rsync --version rsync version 2.6.9 protocol version 29 [EMAIL PROTECTED] cron.daily]# uname -a Linux srv01.wcsl.local 2.6.9-42.0.10.EL #1 Tue Feb 27 09:24:42 EST 2007 i686 i686 i386 GNU/Linux [EMAIL PROTECTED] cron.daily]# Error received when trying to use rsync -vz on the client inflate (token) returned -5 rsync error: error in rsync protocol data stream (code 12) at token.c(458) rsync: writefd_unbuffered failed to write 4092 bytes: phase "unknown" [generator]: Broken pipe (32) rsync error: error in rsync protocol data stream (code 12) at io.c(909) Seems to work fine without compression -z flag omitted. Thanks in advance Regards /Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Files copied into wrong directory
On 7/19/07, Alan Cheers <[EMAIL PROTECTED]> wrote: I think I understand a bit better now. To use SSH you don't get to use the daemon (unless you specify the remote shell/which may cause loss of features). Well, you have the choice of (1) a single-use daemon invoked over ssh or (2) a connection proxied or forwarded by ssh to a background daemon. In general, if you merely want to use an rsync feature that is available only when using a daemon, you would invoke a single-use daemon (#1) in your own remote account. On the other hand, if you want a daemon that authenticates untrusted users and then offers them carefully controlled access, you would usually set up a background daemon (#2) and then add the ssh if you need the encryption or firewall-bypassing. To do #1, all you have to do is move the configuration file on the daemon side if necessary (the daemon looks for it by default in the remote home directory instead of /etc) and pass "-e ssh" on the client. To do #2... I am still interested in encrypting the traffic AND using a daemon. The manual says something to the effect of using SSH to tunnel a local port to a remote machine and configure a normal rsync daemon to accept localhost traffic. Can you explain the tunneling a port part? This would be the fourth method on http://rsync.samba.org/firewall.html . On the client, you tell ssh to forward connections to a local port of your choice to the remote port on which the daemon is listening. That means that, every time something on your computer connects to the local port, the local ssh signals the remote ssh to open a corresponding connection to the daemon's port. The two ssh processes then pass the data back and forth over the encrypted connection so that, for all the rsync client can tell, it is talking directly to the daemon. This setup guarantees that no one can snoop on your own exchange with the daemon but does nothing to stop others from connecting to it themselves and accessing the modules. You can block this in either or both of two places: (1) stop others from connecting to the daemon, or (2) set up authentication on the daemon so they can't access modules once they connect. To do #2, create a secrets file and set "auth users"; see the rsyncd.conf man page. For #1, you're already safe if a firewall blocks connections to the daemon port from outside a private network whose users you all trust. If not, you can tell the daemon to accept connections only from the remote machine itself by setting "address = localhost"; then you're in danger only from users who can log into the remote machine via ssh or similar means and make such connections. Notwithstanding that paragraph, if you forward a local port to the daemon's port, you're additionally in danger of others on the local machine piggybacking on that forward. Using a ProxyCommand in place of a port forward (see method 2 on http://rsync.samba.org/firewall.html ) doesn't have the piggybacking problem and also tends to be more convenient. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files permission and hidden files are not rsync properly
On 7/19/07, Wayne Davison <[EMAIL PROTECTED]> wrote: On Thu, Jul 19, 2007 at 10:42:05AM -0400, Abdul Khan wrote: > rsync --verbose --progress --stats --recursive > /home/vpopmail/ 192.167.1.61::email_backup/ You didn't specify owner or group preservation. I recommend using -a (--archive) so that you get everything preserved and everything copied. Indeed; I missed that. In rsync 3.0.0 (which is not yet released), there is a way to run a non-root daemon that can save privileged information: the --fake-super option will store things like ownership info (etc.) into an extended attribute. This will allow a daemon to run as user nobody without losing the ability to backup and restore the full details of files. Could you make it possible (at least as an option) to store the original permissions in the xattr but then create all files with the default permissions, or possibly the original permissions modulo a chmod option? That way, I could make a snapshot and specify who can read or write the snapshot independently of the permissions represented inside the snapshot. For example, as root, I could back up my machine and offer all users read-only access to retrieve their stuff from the backup, even while some directories (e.g., /tmp) of the backup retain the information that they were originally world-writable. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rysnc Schedule
On 7/19/07, Chris <[EMAIL PROTECTED]> wrote: I know how to use the command prompt, was just curious if it had a gui. OK. Please don't take any offense at my offering a little extra support in case you were a user who needed it. So this can backup for 30 days and once 30 days comes up, it will delete the 1st day and add the 31st? Yes. Just set a backup interval of "daily" with 30 snapshots in the configuration file. If you like, you can also add other intervals, e.g., weekly snapshots for the past 10 weeks. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 4803] link-dest folder is not validated
https://bugzilla.samba.org/show_bug.cgi?id=4803 --- Comment #1 from [EMAIL PROTECTED] 2007-07-19 14:17 CST --- A leading ~ is a shell expansion that only occurs at the start of an arg. If you want the shell to expand it, get rid of the '=' in the --link-dest option: --link-dest ~/Desktop/folder2 Seems like a good idea to make rsync iterate over the --*-dest args and report problems. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 4803] New: link-dest folder is not validated
https://bugzilla.samba.org/show_bug.cgi?id=4803 Summary: link-dest folder is not validated Product: rsync Version: 3.0.0 Platform: Other OS/Version: Mac OS X Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: [EMAIL PROTECTED] ReportedBy: [EMAIL PROTECTED] QAContact: [EMAIL PROTECTED] While I get error messages if the Source or Destination is invalid, I don't get one for an invalid --link-dest . A link-dest of $HOME/... works while ~/... does not. ie rsync -avi --link-dest=$HOME/Desktop/folder2 ~/Desktop/folder1/ ~/Desktop/folder3/ and rsync -avi --link-dest=../Desktop/folder2 ~/Desktop/folder1/ ~/Desktop/folder3/ will both hard link, rsync -avi --link-dest=~/Desktop/folder2 ~/Desktop/folder1/ ~/Desktop/folder3/ does not. The problem is componded by the lack of a warning message -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rysnc Schedule
On Wed, Jul 18, 2007 at 08:45:28PM -0400, Matt McCutchen wrote: > On 7/18/07, Charles Marcus <[EMAIL PROTECTED]> wrote: > >Is there an alternate way to remove an entire directory *without* > >traversing the directory itself? > > No filesystem that I know of provides one, although it would be great > to be able to issue a single unlink or rmdir call and have the > filesystem traverse the directory and free its insides in the > background. You may also wish to consider dirvish at www.dirvish.org . It is a scheduling wrapper around rsync, and includes an expire utility. Keith -- Keith Lofstrom [EMAIL PROTECTED] Voice (503)-520-1993 KLIC --- Keith Lofstrom Integrated Circuits --- "Your Ideas in Silicon" Design Contracting in Bipolar and CMOS - Analog, Digital, and Scan ICs -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files permission and hidden files are not rsync properly
On Thu, Jul 19, 2007 at 10:42:05AM -0400, Abdul Khan wrote: > rsync --verbose --progress --stats --recursive > /home/vpopmail/ 192.167.1.61::email_backup/ You didn't specify owner or group preservation. I recommend using -a (--archive) so that you get everything preserved and everything copied. It sounds like you have already updated the daemon config so that the receiving side is running as root (which makes the preservation of ownership possible). In rsync 3.0.0 (which is not yet released), there is a way to run a non-root daemon that can save privileged information: the --fake-super option will store things like ownership info (etc.) into an extended attribute. This will allow a daemon to run as user nobody without losing the ability to backup and restore the full details of files. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rysnc Schedule
On 7/19/07, Chris <[EMAIL PROTECTED]> wrote: Never used it before. Does it have a gui? No. You edit the configuration file /etc/rsnapshot.conf and then run rsnapshot from the command line with the name of the backup interval, like "rsnapshot daily". Please don't be put off by having to edit a configuration file yourself; it's pretty straightforward. Just go through and replace the samples with the paths and settings that you want to use. There are plenty of comments to guide you. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Files copied into wrong directory
On 7/19/07, Alan Cheers <[EMAIL PROTECTED]> wrote: I've got to be overlooking something but when I use rsync client to copy files from one box to another it creates the folder/files in the root directory instead of what is specified in the module path statement. Did you write the destination with a single colon? If so, replace it with a double colon. A single colon indicates rsync over a remote shell; you have to use a double colon to access an rsync daemon. This is a very common mistake. If that isn't the problem, please send your rsync command line and daemon configuration file so I have a clue what your setup is. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Files copied into wrong directory
I've got to be overlooking something but when I use rsync client to copy files from one box to another it creates the folder/files in the root directory instead of what is specified in the module path statement. I have tried it via ssh and without. With user authentication on and without. I also stripped down the daemon config file to the bare minimum. Still no luck. Can somebody explain to me what I am doing wrong? What is the relationship between rsync daemon and the path? What would prevent it from using the specified path? Thanks for any tips. -Alan -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files permission and hidden files are not rsync properly
> In the future and if you need more help on this issue than the below, > please send the exact rsync command you used! Here is the command that I ran from the destinition machine which is also running the rsync daemon or the rsync server; [EMAIL PROTECTED] admin]# rsync --verbose --progress --stats --recursive /home/vpopmail/ 192.167.1.61::email_backup/ they are not symlinks as log mentioned they are hidden files (or dot files), "Hidden file" is not a file type. On Unix-like systems, the concept of hidden-ness does not exist at the filesystem level or in most filesystem tools (e.g., rsync). That a file whose name begins with a dot is considered hidden is merely a convention among the user interfaces of most file listers/browsers and a few other tools. so is there any thing specific that I have to add in the command to copy those as well? As per the above, dot files are nothing out of the ordinary to rsync and no special option is needed to make rsync copy them. >Please send the "ls -la" output showing the dot files so I can see >what their file type is (as indicated by the first character of each >line of the listing). Matt you are right that the hidden files are not file types. I certainly overlooked at them and they are basically Symlinks. (sorry for my previous claim) I was basically mixing with one of our other box that I had to create some hidden files (dot files) manually. Applology. Since that is out of the way I can pass --links with rsync command to rsync the symlinks. Here is the ls -la output of few of the files that you asked for; lrwxrwxrwx 1 vpopmail vchkpw57 Oct 23 2006 .qmail-allusers -> /home/vpopmail/domains/f leetwoodmetal.com/allusers/editor lrwxrwxrwx 1 vpopmail vchkpw60 Oct 23 2006 .qmail-allusers-accept-default -> /home/vpo pmail/domains/fleetwoodmetal.com/allusers/moderator lrwxrwxrwx 1 vpopmail vchkpw58 Oct 23 2006 .qmail-allusers-default -> /home/vpopmail/d omains/fleetwoodmetal.com/allusers/manager >I just realized a much more likely cause of the problem. If the rsync >daemon on the destination is started as "root", by default it changes >IDs to "nobody:nobody" before accessing the module and thus loses the >power to set the owners of received files arbitrarily (in this case, >to "vpopmail:vchkpw"). To make the daemon stay as root so it can >chown the files, explicitly say "uid = root" and "gid = root" in the >rsyncd.conf . Here is the existing /etc/rsyncd.conf file; [EMAIL PROTECTED] fleetwoodmetal.com]# cat /etc/rsyncd.conf log file = /var/log/rsyncd.log log format = %h %o %f %l %b [email_backup] path = /email_backup comment = email backup list = yes read only = no Which I am going to change it to ; log file = /var/log/rsyncd.log log format = %h %o %f %l %b uid = root gid = root [email_backup] path = /email_backup comment = email backup list = yes read only = no Thank you for your help again. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files permission and hidden files are not rsync properly
On 7/19/07, Abdul Khan <[EMAIL PROTECTED]> wrote: > In the future and if you need more help on this issue than the below, > please send the exact rsync command you used! Here is the command that I ran from the destinition machine which is also running the rsync daemon or the rsync server; [EMAIL PROTECTED] admin]# rsync --verbose --progress --stats --recursive /home/vpopmail/ 192.167.1.61::email_backup/ they are not symlinks as log mentioned they are hidden files (or dot files), "Hidden file" is not a file type. On Unix-like systems, the concept of hidden-ness does not exist at the filesystem level or in most filesystem tools (e.g., rsync). That a file whose name begins with a dot is considered hidden is merely a convention among the user interfaces of most file listers/browsers and a few other tools. so is there any thing specific that I have to add in the command to copy those as well? As per the above, dot files are nothing out of the ordinary to rsync and no special option is needed to make rsync copy them. Please send the "ls -la" output showing the dot files so I can see what their file type is (as indicated by the first character of each line of the listing). > By default, rsync preserves owners and groups by name if possible or > otherwise by number. That means if the destination machine lacks a > "vpopmail" user and a "vchkpw" group, rsync sets the same numerical > IDs on the destination as on the source. If those IDs represent > "nobody" and "nobody" on the destination, the files get that > ownership. Creating "vpopmail" and "vchkpw" on the destination > machine should fix the problem. The destination machine is actually the mirror of the source so both mochines are identical and both "vpopmail" user and "vchkpw" group do exist on the destinition. here is the evidence of "vpopmail" and "vchpkw" on the destinition; [EMAIL PROTECTED] admin]# cat /etc/group | grep vchk* vchkpw:x:89: [EMAIL PROTECTED] admin]# cat /etc/passwd | grep vpopmail vpopmail:x:89:89::/home/vpopmail:/bin/bash I just realized a much more likely cause of the problem. If the rsync daemon on the destination is started as "root", by default it changes IDs to "nobody:nobody" before accessing the module and thus loses the power to set the owners of received files arbitrarily (in this case, to "vpopmail:vchkpw"). To make the daemon stay as root so it can chown the files, explicitly say "uid = root" and "gid = root" in the rsyncd.conf . Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files permission and hidden files are not rsync properly
On 7/19/07, Abdul Khan <[EMAIL PROTECTED]> wrote: >> I have recently installed rsync.2.6.9. on CentOS4.x boxes. That was my >> first backup yesterday. In the future and if you need more help on this issue than the below, please send the exact rsync command you used! >> But I notice that some hidden (or dot files) are not copied to the other >> host on the network. Rsync said "skipping non-regular file". That means the files aren't regular and you haven't passed the option to tell rsync to copy files of that type. What are those files? Symlinks? If so, then pass -l. >> And group and ownership of the files are changed to >> "nobody" while files and folders permission are copied properly. So what >> am mission here can anybody shed light please? By default, rsync preserves owners and groups by name if possible or otherwise by number. That means if the destination machine lacks a "vpopmail" user and a "vchkpw" group, rsync sets the same numerical IDs on the destination as on the source. If those IDs represent "nobody" and "nobody" on the destination, the files get that ownership. Creating "vpopmail" and "vchkpw" on the destination machine should fix the problem. Matt Hi Matt, Thank you for response. > In the future and if you need more help on this issue than the below, > please send the exact rsync command you used! Here is the command that I ran from the destinition machine which is also running the rsync daemon or the rsync server; [EMAIL PROTECTED] admin]# rsync --verbose --progress --stats --recursive /home/vpopmail/ 192.167.1.61::email_backup/ they are not symlinks as log mentioned they are hidden files (or dot files), so is there any thing specific that I have to add in the command to copy those as well? > By default, rsync preserves owners and groups by name if possible or > otherwise by number. That means if the destination machine lacks a > "vpopmail" user and a "vchkpw" group, rsync sets the same numerical > IDs on the destination as on the source. If those IDs represent > "nobody" and "nobody" on the destination, the files get that > ownership. Creating "vpopmail" and "vchkpw" on the destination > machine should fix the problem. The destination machine is actually the mirror of the source so both mochines are identical and both "vpopmail" user and "vchkpw" group do exist on the destinition. here is the evidence of "vpopmail" and "vchpkw" on the destinition; [EMAIL PROTECTED] admin]# cat /etc/group | grep vchk* vchkpw:x:89: [EMAIL PROTECTED] admin]# cat /etc/passwd | grep vpopmail vpopmail:x:89:89::/home/vpopmail:/bin/bash Thanks in advance. ak -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Local disk to disk Rsync taking an hour longer than disk to remote
Sincere thanks to everyone for their replies. Before following some of the advice offered I decided to investigate the machine filesystem further. I'm no linux expert (obviously :)) but found that of the ext3 filesystems on each disk, the dir_index feature was only enabled on the primary disk. Research revealed that this helped with directory listing and it made sense that with 1mill files in one directory, it might be a good idea to have it enabled on both disks! Result: The local backup completes in <10min instead of 1hr20min+. It also explains why local primary disk to remote primary disk backup was fast, as both these had dir_index enabled. On CentOS at least, additional mounted ext3 drives don't seem to have dir_index enabled by default. It's a massive relief to have this finally sorted, I've spent far too many hours trying to fix it. I don't know whether this might be a useful tip in the Rsync FAQ troubleshooting when backing up large directories of small files. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rysnc Schedule
On Thu, 2007-07-19 at 09:30 -0400, Matt McCutchen wrote:
> On 7/19/07, Rick Romero <[EMAIL PROTECTED]> wrote:
> > /usr/bin/find /var/backup/* -mtime +30 -exec rm -rf {} \;
> >
> > Anything older than 30 days gets deleted from /var/backup/
>
> That's no good: it deletes individual files that happen to be older
> than 30 days from even yesterday's backup. This is better:
>
> find /var/backup -maxdepth 1 -mtime +30 -exec rm -rf {} \;
Oh of course - that's what I get for copying an existing cronjob and not
thinking it through.
> Sparty2809, are you sure you don't want to use rsnapshot? It does all
> of this for you.
I very well may use rsnapshot - I've been putting off trying to do a
rotation like that on my own, it looks like a great solution.
Rick
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files permission and hidden files are not rsync properly
On 7/19/07, Abdul Khan <[EMAIL PROTECTED]> wrote: I have recently installed rsync.2.6.9. on CentOS4.x boxes. That was my first backup yesterday. In the future and if you need more help on this issue than the below, please send the exact rsync command you used! But I notice that some hidden (or dot files) are not copied to the other host on the network. Rsync said "skipping non-regular file". That means the files aren't regular and you haven't passed the option to tell rsync to copy files of that type. What are those files? Symlinks? If so, then pass -l. And group and ownership of the files are changed to "nobody" while files and folders permission are copied properly. So what am mission here can anybody shed light please? By default, rsync preserves owners and groups by name if possible or otherwise by number. That means if the destination machine lacks a "vpopmail" user and a "vchkpw" group, rsync sets the same numerical IDs on the destination as on the source. If those IDs represent "nobody" and "nobody" on the destination, the files get that ownership. Creating "vpopmail" and "vchkpw" on the destination machine should fix the problem. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rysnc Schedule
On 7/19/07, Rick Romero <[EMAIL PROTECTED]> wrote:
/usr/bin/find /var/backup/* -mtime +30 -exec rm -rf {} \;
Anything older than 30 days gets deleted from /var/backup/
That's no good: it deletes individual files that happen to be older
than 30 days from even yesterday's backup. This is better:
find /var/backup -maxdepth 1 -mtime +30 -exec rm -rf {} \;
Sparty2809, are you sure you don't want to use rsnapshot? It does all
of this for you.
Matt
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
files permission and hidden files are not rsync properly
Hi folks, I have recently installed rsync.2.6.9. on CentOS4.x boxes. That was my first backup yesterday. But I notice that some hidden (or dot files) are not copied to the other host on the network. And group and ownership of the files are changed to “nobody” while files and folders permission are copied properly. So what am mission here can anybody shed light please? Here is the result; --- Number of files: 101558 Number of files transferred: 94209 Total file size: 21140359756 bytes Total transferred file size: 21140357580 bytes Literal data: 21063075446 bytes Matched data: 0 bytes File list size: 7819534 File list generation time: 62.368 seconds File list transfer time: 0.000 seconds Total bytes sent: 21077482260 Total bytes received: 1884192 sent 21077482260 bytes received 1884192 bytes 3231792.48 bytes/sec total size is 21140359756 speedup is 1.00 rsync warning: some files vanished before they could be transferred (code 24) at main.c(977) [sender=2.6.9] - And here is the log; - [EMAIL PROTECTED] email_backup]# tail /var/log/rsyncd.log 2007/07/18 19:03:19 [5675] skipping non-regular file "domains/fleetwoodmetal.com/.qmail-windsorusers-return-default" 2007/07/18 19:17:37 [5675] skipping non-regular file "domains/fleetwoodmetal.com/0/tinag/IMAPdir/INBOX" 2007/07/18 19:27:49 [5675] skipping non-regular file "domains/fleetwoodmetal.com/akhan/IMAPdir/INBOX" 2007/07/18 20:39:18 [5675] skipping non-regular file "domains/fleetwoodmetal.com/lisat/IMAPdir/INBOX" 2007/07/18 20:43:43 [5675] skipping non-regular file "domains/fleetwoodmetal.com/paulc/IMAPdir/INBOX" 2007/07/18 20:46:53 [5675] skipping non-regular file "domains/systems.fleetwoodmetal.com/.qmail-fmi" 2007/07/18 20:47:13 [5675] skipping non-regular file "domains/systems.fleetwoodmetal.com/.qmail-fmi-default" 2007/07/18 20:47:13 [5675] skipping non-regular file "domains/systems.fleetwoodmetal.com/.qmail-fmi-owner" 2007/07/18 20:47:13 [5675] skipping non-regular file "domains/systems.fleetwoodmetal.com/.qmail-fmi-return-default" 2007/07/18 20:50:58 [5675] wrote 28 bytes read 21077482315 bytes total size 21140357580 -- Permission, ownership and group on the source box (before rsync) drwx-- 38 vpopmail vchkpw 4096 Jul 9 13:58 0 drwx-- 6 vpopmail vchkpw 4096 Sep 18 2006 akhan drwx-- 3 vpopmail vchkpw 4096 Feb 23 11:09 alb drwx-- 8 vpopmail vchkpw 4096 Jun 27 13:37 allusers drwx-- 3 vpopmail vchkpw 4096 Sep 18 2006 alm drwx-- 3 vpopmail vchkpw 4096 Feb 22 08:17 angilad drwx-- 3 vpopmail vchkpw 4096 Jul 4 11:48 barbm drwx-- 3 vpopmail vchkpw 4096 Jul 16 08:00 billj drwx-- 3 vpopmail vchkpw 4096 Apr 10 11:04 bills drwx-- 3 vpopmail vchkpw 4096 Sep 18 2006 bonitap drwx-- 3 vpopmail vchkpw 4096 Feb 23 11:07 brianm - Permission, ownership and group after rsync - drwx-- 38 nobody nobody 4096 Jul 18 15:27 0 drwx-- 6 nobody nobody 4096 Jul 18 15:34 akhan drwx-- 3 nobody nobody 4096 Jul 18 15:35 alb drwx-- 8 nobody nobody 4096 Jul 18 15:35 allusers drwx-- 3 nobody nobody 4096 Jul 18 15:35 alm drwx-- 3 nobody nobody 4096 Jul 18 15:35 angilad drwx-- 3 nobody nobody 4096 Jul 18 15:36 barbm drwx-- 3 nobody nobody 4096 Jul 18 15:37 billj drwx-- 3 nobody nobody 4096 Jul 18 15:37 bills drwx-- 3 nobody nobody 4096 Jul 18 15:38 bonitap drwx-- 3 nobody nobody 4096 Jul 18 15:38 brianm -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rysnc Schedule
/usr/bin/find /var/backup/* -mtime +30 -exec rm -rf {} \;
Anything older than 30 days gets deleted from /var/backup/
On Thu, 2007-07-19 at 05:46 -0700, sparty2809 wrote:
> I guess I should have been clearer. I know how to delete files/directories.
> I am just not sure how to delete say the 31st and so forth.
>
>
>
> Chuck Wolber wrote:
> >
> > On Wed, 18 Jul 2007, sparty2809 wrote:
> >
> >> I understand how to do the rsync part, but how would I delete the files?
> >
> > In linux, you delete files with the "rm" command... In windows you'd
> > probably want to use "deltree"... Or, you could get really fancy and use
> > directed charges of thermite, although that might make it difficult to
> > re-use portions of the disk...
> >
> > ..Chuck..
> >
> >
> > --
> > http://www.quantumlinux.com
> > Quantum Linux Laboratories, LLC.
> > ACCELERATING Business with Open Technology
> >
> > "The measure of the restoration lies in the extent to which we apply
> > social values more noble than mere monetary profit." - FDR
> >
> > 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
> > --
> > To unsubscribe or change options:
> > https://lists.samba.org/mailman/listinfo/rsync
> > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Rysnc-Schedule-tf4104577.html#a11688212
> Sent from the Samba - rsync mailing list archive at Nabble.com.
>
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync Schedule
> On 7/18/2007, Chuck Wolber ([EMAIL PROTECTED]) wrote:
> > In linux, you delete files with the "rm" command...
>
> OT follow-up...
>
> Is there an alternate way to remove an entire directory *without*
> traversing the directory itself?
I just use rmdir under Windows.
Also Windows users can use :
FORFILES [/P pathname] [/M searchmask] [/S]
[/C command] [/D [+ | -] {dd/MM/ | dd}]
Description:
Selects a file (or set of files) and executes a
command on that file. This is helpful for batch jobs.
for example:
set drive=E:\
forfiles /p %drive%:\ /m * /D -30 /C "cmd /c if @isdir==TRUE ECHO @PATH>>
C:\list.tmp"
This lists directories over 30 days old into a file.
I make the script delete the old directories and it loops until there is
no old directories left.
This email is the property of ECS Technology Ltd.
This company is registered in Scotland with company number 212513.
VAT registered GB 761 7456 12
http://www.ecs-tech.com/
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rysnc Schedule
I guess I should have been clearer. I know how to delete files/directories. I am just not sure how to delete say the 31st and so forth. Chuck Wolber wrote: > > On Wed, 18 Jul 2007, sparty2809 wrote: > >> I understand how to do the rsync part, but how would I delete the files? > > In linux, you delete files with the "rm" command... In windows you'd > probably want to use "deltree"... Or, you could get really fancy and use > directed charges of thermite, although that might make it difficult to > re-use portions of the disk... > > ..Chuck.. > > > -- > http://www.quantumlinux.com > Quantum Linux Laboratories, LLC. > ACCELERATING Business with Open Technology > > "The measure of the restoration lies in the extent to which we apply > social values more noble than mere monetary profit." - FDR > > 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0 > -- > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > > -- View this message in context: http://www.nabble.com/Rysnc-Schedule-tf4104577.html#a11688212 Sent from the Samba - rsync mailing list archive at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Auto Reply
James Dishongh is not available. Please forward any issues to [EMAIL PROTECTED], or call 360-567-0469. Thank you, Net-Rx Support Team -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
