Re: files permission and hidden files are not rsync properly
On 7/20/07, Abdul Khan [EMAIL PROTECTED] wrote: Thanks for the reply, the permission issue is resolved after I reconfigured and reinstalled rsync. The only issue I have now is user vpopmail and group vchkpw is not retaining. Its just not retaining the user and the group of files. I even tried to explicitly put uid=root and gid=root in the /etc/rsync.conf file but it does not work. Did you add something to the client's command line to tell it to preserve users and groups (either -og, or -a to preserve everything as Wayne suggested)? If so, what users and groups are being set on the destination? Still nobody:nobody? Another question: i have to manually put list's email address rsync@lists.samba.org to send mail to the list instead of just hit the reply. I mean it should go to the list instead of any member. Can anybody just guide me? If your email client has a Reply to All, use that instead of Reply. Reply replies only to the From address of the original message, while Reply to All also sends a copy to everyone else who was listed in the To or Cc of the original message. The rsync@lists.samba.org address, since it never sends messages itself, would be among these everyone else. If there's no Reply to All, get a better email client or create a separate webmail account (e.g., Gmail ;) ) for communications with the rsync list. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
files permission and hidden files are not rsync properly
Hi folks, I have recently installed rsync.2.6.9. on CentOS4.x boxes. That was my first backup yesterday. But I notice that some hidden (or dot files) are not copied to the other host on the network. And group and ownership of the files are changed to “nobody” while files and folders permission are copied properly. So what am mission here can anybody shed light please? Here is the result; --- Number of files: 101558 Number of files transferred: 94209 Total file size: 21140359756 bytes Total transferred file size: 21140357580 bytes Literal data: 21063075446 bytes Matched data: 0 bytes File list size: 7819534 File list generation time: 62.368 seconds File list transfer time: 0.000 seconds Total bytes sent: 21077482260 Total bytes received: 1884192 sent 21077482260 bytes received 1884192 bytes 3231792.48 bytes/sec total size is 21140359756 speedup is 1.00 rsync warning: some files vanished before they could be transferred (code 24) at main.c(977) [sender=2.6.9] - And here is the log; - [EMAIL PROTECTED] email_backup]# tail /var/log/rsyncd.log 2007/07/18 19:03:19 [5675] skipping non-regular file domains/fleetwoodmetal.com/.qmail-windsorusers-return-default 2007/07/18 19:17:37 [5675] skipping non-regular file domains/fleetwoodmetal.com/0/tinag/IMAPdir/INBOX 2007/07/18 19:27:49 [5675] skipping non-regular file domains/fleetwoodmetal.com/akhan/IMAPdir/INBOX 2007/07/18 20:39:18 [5675] skipping non-regular file domains/fleetwoodmetal.com/lisat/IMAPdir/INBOX 2007/07/18 20:43:43 [5675] skipping non-regular file domains/fleetwoodmetal.com/paulc/IMAPdir/INBOX 2007/07/18 20:46:53 [5675] skipping non-regular file domains/systems.fleetwoodmetal.com/.qmail-fmi 2007/07/18 20:47:13 [5675] skipping non-regular file domains/systems.fleetwoodmetal.com/.qmail-fmi-default 2007/07/18 20:47:13 [5675] skipping non-regular file domains/systems.fleetwoodmetal.com/.qmail-fmi-owner 2007/07/18 20:47:13 [5675] skipping non-regular file domains/systems.fleetwoodmetal.com/.qmail-fmi-return-default 2007/07/18 20:50:58 [5675] wrote 28 bytes read 21077482315 bytes total size 21140357580 -- Permission, ownership and group on the source box (before rsync) drwx-- 38 vpopmail vchkpw 4096 Jul 9 13:58 0 drwx-- 6 vpopmail vchkpw 4096 Sep 18 2006 akhan drwx-- 3 vpopmail vchkpw 4096 Feb 23 11:09 alb drwx-- 8 vpopmail vchkpw 4096 Jun 27 13:37 allusers drwx-- 3 vpopmail vchkpw 4096 Sep 18 2006 alm drwx-- 3 vpopmail vchkpw 4096 Feb 22 08:17 angilad drwx-- 3 vpopmail vchkpw 4096 Jul 4 11:48 barbm drwx-- 3 vpopmail vchkpw 4096 Jul 16 08:00 billj drwx-- 3 vpopmail vchkpw 4096 Apr 10 11:04 bills drwx-- 3 vpopmail vchkpw 4096 Sep 18 2006 bonitap drwx-- 3 vpopmail vchkpw 4096 Feb 23 11:07 brianm - Permission, ownership and group after rsync - drwx-- 38 nobody nobody 4096 Jul 18 15:27 0 drwx-- 6 nobody nobody 4096 Jul 18 15:34 akhan drwx-- 3 nobody nobody 4096 Jul 18 15:35 alb drwx-- 8 nobody nobody 4096 Jul 18 15:35 allusers drwx-- 3 nobody nobody 4096 Jul 18 15:35 alm drwx-- 3 nobody nobody 4096 Jul 18 15:35 angilad drwx-- 3 nobody nobody 4096 Jul 18 15:36 barbm drwx-- 3 nobody nobody 4096 Jul 18 15:37 billj drwx-- 3 nobody nobody 4096 Jul 18 15:37 bills drwx-- 3 nobody nobody 4096 Jul 18 15:38 bonitap drwx-- 3 nobody nobody 4096 Jul 18 15:38 brianm -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files permission and hidden files are not rsync properly
On 7/19/07, Abdul Khan [EMAIL PROTECTED] wrote: I have recently installed rsync.2.6.9. on CentOS4.x boxes. That was my first backup yesterday. In the future and if you need more help on this issue than the below, please send the exact rsync command you used! But I notice that some hidden (or dot files) are not copied to the other host on the network. Rsync said skipping non-regular file. That means the files aren't regular and you haven't passed the option to tell rsync to copy files of that type. What are those files? Symlinks? If so, then pass -l. And group and ownership of the files are changed to nobody while files and folders permission are copied properly. So what am mission here can anybody shed light please? By default, rsync preserves owners and groups by name if possible or otherwise by number. That means if the destination machine lacks a vpopmail user and a vchkpw group, rsync sets the same numerical IDs on the destination as on the source. If those IDs represent nobody and nobody on the destination, the files get that ownership. Creating vpopmail and vchkpw on the destination machine should fix the problem. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files permission and hidden files are not rsync properly
On 7/19/07, Abdul Khan [EMAIL PROTECTED] wrote: I have recently installed rsync.2.6.9. on CentOS4.x boxes. That was my first backup yesterday. In the future and if you need more help on this issue than the below, please send the exact rsync command you used! But I notice that some hidden (or dot files) are not copied to the other host on the network. Rsync said skipping non-regular file. That means the files aren't regular and you haven't passed the option to tell rsync to copy files of that type. What are those files? Symlinks? If so, then pass -l. And group and ownership of the files are changed to nobody while files and folders permission are copied properly. So what am mission here can anybody shed light please? By default, rsync preserves owners and groups by name if possible or otherwise by number. That means if the destination machine lacks a vpopmail user and a vchkpw group, rsync sets the same numerical IDs on the destination as on the source. If those IDs represent nobody and nobody on the destination, the files get that ownership. Creating vpopmail and vchkpw on the destination machine should fix the problem. Matt Hi Matt, Thank you for response. In the future and if you need more help on this issue than the below, please send the exact rsync command you used! Here is the command that I ran from the destinition machine which is also running the rsync daemon or the rsync server; [EMAIL PROTECTED] admin]# rsync --verbose --progress --stats --recursive /home/vpopmail/ 192.167.1.61::email_backup/ they are not symlinks as log mentioned they are hidden files (or dot files), so is there any thing specific that I have to add in the command to copy those as well? By default, rsync preserves owners and groups by name if possible or otherwise by number. That means if the destination machine lacks a vpopmail user and a vchkpw group, rsync sets the same numerical IDs on the destination as on the source. If those IDs represent nobody and nobody on the destination, the files get that ownership. Creating vpopmail and vchkpw on the destination machine should fix the problem. The destination machine is actually the mirror of the source so both mochines are identical and both vpopmail user and vchkpw group do exist on the destinition. here is the evidence of vpopmail and vchpkw on the destinition; [EMAIL PROTECTED] admin]# cat /etc/group | grep vchk* vchkpw:x:89: [EMAIL PROTECTED] admin]# cat /etc/passwd | grep vpopmail vpopmail:x:89:89::/home/vpopmail:/bin/bash Thanks in advance. ak -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files permission and hidden files are not rsync properly
On 7/19/07, Abdul Khan [EMAIL PROTECTED] wrote: In the future and if you need more help on this issue than the below, please send the exact rsync command you used! Here is the command that I ran from the destinition machine which is also running the rsync daemon or the rsync server; [EMAIL PROTECTED] admin]# rsync --verbose --progress --stats --recursive /home/vpopmail/ 192.167.1.61::email_backup/ they are not symlinks as log mentioned they are hidden files (or dot files), Hidden file is not a file type. On Unix-like systems, the concept of hidden-ness does not exist at the filesystem level or in most filesystem tools (e.g., rsync). That a file whose name begins with a dot is considered hidden is merely a convention among the user interfaces of most file listers/browsers and a few other tools. so is there any thing specific that I have to add in the command to copy those as well? As per the above, dot files are nothing out of the ordinary to rsync and no special option is needed to make rsync copy them. Please send the ls -la output showing the dot files so I can see what their file type is (as indicated by the first character of each line of the listing). By default, rsync preserves owners and groups by name if possible or otherwise by number. That means if the destination machine lacks a vpopmail user and a vchkpw group, rsync sets the same numerical IDs on the destination as on the source. If those IDs represent nobody and nobody on the destination, the files get that ownership. Creating vpopmail and vchkpw on the destination machine should fix the problem. The destination machine is actually the mirror of the source so both mochines are identical and both vpopmail user and vchkpw group do exist on the destinition. here is the evidence of vpopmail and vchpkw on the destinition; [EMAIL PROTECTED] admin]# cat /etc/group | grep vchk* vchkpw:x:89: [EMAIL PROTECTED] admin]# cat /etc/passwd | grep vpopmail vpopmail:x:89:89::/home/vpopmail:/bin/bash I just realized a much more likely cause of the problem. If the rsync daemon on the destination is started as root, by default it changes IDs to nobody:nobody before accessing the module and thus loses the power to set the owners of received files arbitrarily (in this case, to vpopmail:vchkpw). To make the daemon stay as root so it can chown the files, explicitly say uid = root and gid = root in the rsyncd.conf . Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files permission and hidden files are not rsync properly
In the future and if you need more help on this issue than the below, please send the exact rsync command you used! Here is the command that I ran from the destinition machine which is also running the rsync daemon or the rsync server; [EMAIL PROTECTED] admin]# rsync --verbose --progress --stats --recursive /home/vpopmail/ 192.167.1.61::email_backup/ they are not symlinks as log mentioned they are hidden files (or dot files), Hidden file is not a file type. On Unix-like systems, the concept of hidden-ness does not exist at the filesystem level or in most filesystem tools (e.g., rsync). That a file whose name begins with a dot is considered hidden is merely a convention among the user interfaces of most file listers/browsers and a few other tools. so is there any thing specific that I have to add in the command to copy those as well? As per the above, dot files are nothing out of the ordinary to rsync and no special option is needed to make rsync copy them. Please send the ls -la output showing the dot files so I can see what their file type is (as indicated by the first character of each line of the listing). Matt you are right that the hidden files are not file types. I certainly overlooked at them and they are basically Symlinks. (sorry for my previous claim) I was basically mixing with one of our other box that I had to create some hidden files (dot files) manually. Applology. Since that is out of the way I can pass --links with rsync command to rsync the symlinks. Here is the ls -la output of few of the files that you asked for; lrwxrwxrwx 1 vpopmail vchkpw57 Oct 23 2006 .qmail-allusers - /home/vpopmail/domains/f leetwoodmetal.com/allusers/editor lrwxrwxrwx 1 vpopmail vchkpw60 Oct 23 2006 .qmail-allusers-accept-default - /home/vpo pmail/domains/fleetwoodmetal.com/allusers/moderator lrwxrwxrwx 1 vpopmail vchkpw58 Oct 23 2006 .qmail-allusers-default - /home/vpopmail/d omains/fleetwoodmetal.com/allusers/manager I just realized a much more likely cause of the problem. If the rsync daemon on the destination is started as root, by default it changes IDs to nobody:nobody before accessing the module and thus loses the power to set the owners of received files arbitrarily (in this case, to vpopmail:vchkpw). To make the daemon stay as root so it can chown the files, explicitly say uid = root and gid = root in the rsyncd.conf . Here is the existing /etc/rsyncd.conf file; [EMAIL PROTECTED] fleetwoodmetal.com]# cat /etc/rsyncd.conf log file = /var/log/rsyncd.log log format = %h %o %f %l %b [email_backup] path = /email_backup comment = email backup list = yes read only = no Which I am going to change it to ; log file = /var/log/rsyncd.log log format = %h %o %f %l %b uid = root gid = root [email_backup] path = /email_backup comment = email backup list = yes read only = no Thank you for your help again. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files permission and hidden files are not rsync properly
On Thu, Jul 19, 2007 at 10:42:05AM -0400, Abdul Khan wrote: rsync --verbose --progress --stats --recursive /home/vpopmail/ 192.167.1.61::email_backup/ You didn't specify owner or group preservation. I recommend using -a (--archive) so that you get everything preserved and everything copied. It sounds like you have already updated the daemon config so that the receiving side is running as root (which makes the preservation of ownership possible). In rsync 3.0.0 (which is not yet released), there is a way to run a non-root daemon that can save privileged information: the --fake-super option will store things like ownership info (etc.) into an extended attribute. This will allow a daemon to run as user nobody without losing the ability to backup and restore the full details of files. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: files permission and hidden files are not rsync properly
On 7/19/07, Wayne Davison [EMAIL PROTECTED] wrote: On Thu, Jul 19, 2007 at 10:42:05AM -0400, Abdul Khan wrote: rsync --verbose --progress --stats --recursive /home/vpopmail/ 192.167.1.61::email_backup/ You didn't specify owner or group preservation. I recommend using -a (--archive) so that you get everything preserved and everything copied. Indeed; I missed that. In rsync 3.0.0 (which is not yet released), there is a way to run a non-root daemon that can save privileged information: the --fake-super option will store things like ownership info (etc.) into an extended attribute. This will allow a daemon to run as user nobody without losing the ability to backup and restore the full details of files. Could you make it possible (at least as an option) to store the original permissions in the xattr but then create all files with the default permissions, or possibly the original permissions modulo a chmod option? That way, I could make a snapshot and specify who can read or write the snapshot independently of the permissions represented inside the snapshot. For example, as root, I could back up my machine and offer all users read-only access to retrieve their stuff from the backup, even while some directories (e.g., /tmp) of the backup retain the information that they were originally world-writable. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
