RE: Time rsYnc Machine (tym)
M. Carrasco wrote (Friday, August 10, 2012 12:23 AM): Reading the rsync man page, it seems that the -H option with --link-dest is tricky. The price to pay on not using -H is that hard linked files are treated as separated files: I could not find any mechanism in rsync to improve the time machine effect; I would appreciate hits on how to improve it. I highlight this in the tym man page and inside the program. I'm still using the way before -H and --link-dest were available: cp --archive --link $PREVIOUS_SNAPSHOT $NEW_SNAPSHOT rsync \ --archive --verbose --delete\ --delete-excluded --one-file-system \ $SOME_EXCLUDE_INCLUDE_RULES \ $DIR_TO_BACKUP/ $NEW_SNAPSHOT With a bit shuffling around the snapshot directories, I keep 24x hourly, 7x daily, 4x weekly and 12 montly snapshots (using 4 cron jobs). Furthermore, I've mounted the backup partition as /root/backup, with a read-only bind-mount on /media/backup so that the user can always have a safe look into all the snapshots. ;-) Have a nice day, Berny -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: cannot rsync when source directory lacks write permission
Steven, Of course, the destination side must have enough rights to achieve what I need, and an rsyncd running as root:root surely have all the necessary rights. So using --perms with this daemon must have been sufficient. But not, it is simply not true, because the daemon has a built-in assumption which overkills it. I know rsync sends its command line parameters to the daemon, so rsyncd could have been able to handle the case, but instead of recognizing the flag it simply makes that assumption. I think all the permission/ownership handling is complicated (unaccountable, puzzling, peculiar) and the usage is confusing and annoying, it should be reworked in a much more consistent way: 1. use exactly the same options on both sides 2. specify which attributes to transfer (keep) and which ones to set explicitly (including permissions, ownership, time and probably acl also) 3. define defaults 4. define a precedence (like: source filesystem, sender config, receiver config, receiver user rights, defaults) 5. describe actions taken in case of insufficient rights (just another example: client side -E, server side incoming chmod u+x,g-x and outgoing chmod u+x,g-x: what is the expected result when sending or receiving files?) Using aliases these can be mapped to the actual flags - for backward compatibility Andras -Original Message- From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] On Behalf Of Steven Levine Sent: Thursday, August 09, 2012 21:37 To: rsync@lists.samba.org Subject: RE: cannot rsync when source directory lacks write permission In 64fab8215d47a944abdf7de50a3406a219c2dac...@esesscms0353.eemea.ericsson.se, on 08/09/12 at 07:54 AM, András Porjesz andras.porj...@ericsson.com said: Hi, Thanks, it looks ok, just it is not documented anywhere: From the ryncd.conf man page uidThis parameter specifies the user name or user ID that file transfers to and from that module should take place as when the daemon was run as root. In combination with the gid parameter this determines what file permissions are available. The default is uid -2, which is normally the user nobody. it overwrites the -perms flag on the other side. Not really. Se below. So read documentation, it is definitely against it: In summary: to give destination files (both old and new) the source permissions, use --perms. I assume you are referring to the rsync man page which says -p, --perms This option causes the receiving rsync to set the destination permissions to be the same as the source permissions. (See also the --chmod option for a way to modify what rsync considers to be the source permissions.) What it does not say is that the receiving side needs sufficient permissions to be able to change the permissions. I guess the man page authors assumed that someone running rsync on *ix would understand this implicitly. Running the receiving side as root is one option for ensuring that the receiver can change permissions. There are others that are more secure. Running the receiver as the default nobody user does not turn off --perms, it simply ensures that the attempt to change permissions is very likely to fail. One way to make it not fail is to have the module root owned by nobody. Regards, Steven -- -- Steven Levine stev...@earthlink.net eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: cannot rsync when source directory lacks write permission
András Porjesz wrote (Friday, August 10, 2012 9:27 AM): I think all the permission/ownership handling is complicated (unaccountable, puzzling, peculiar) and the usage is confusing and annoying I think you didn't get Steven's point: [...] The default is uid -2, which is normally the user nobody. I.e. even if your rsyncd is running as root, it drops it's priviledges by default by becoming 'nobody'. That is a pre-cautious security issue which meets the requirements for most installations which just serve files read-only. When you want to push things there, then you might either rsync-via-ssh to that server as root (or sudo there), or set 'uid' to root in rsyncd.conf. Have a nice day, Berny -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Time rsYnc Machine (tym)
On Thu 09 Aug 2012, Linda Walsh wrote: Anyway, thanks for the history update. I have a feeling rsync is afraid to use memory -- and really, it should try to use alot of memory to optimize transfers, I have had rsync fail after using up 8GB memory + 4GB swap, so I'm very happy it does its best to minimize memory usage. Turning off partial-send's (--whole-file) using 8-bit-io) seem to really help speed things up on a same-system copy... In doing a full sync with a backup, --whole-file is the default when source and destination are both on the same system -- at least, if you don't specify hostnames... of a 6G HD, (I used --drop-cache, --inplace and --del as well) Doing an archive diff with --acls --xattrs + --hardlinks rysync averaged 125MB/s for the actual IO... (about 30% of the disk)... OK, and now on a 6*TB* HD :-) This is on one of my backup servers: /dev/sdc 39T 33T 5.9T 85% /backup Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: cannot rsync when source directory lacks write permission
Hi Berny, Yes, it is a possible solution, I missed something. But from my point of view you did not get my point. I worked with unix/linux for 20 years therefore I know what are you talking about. In some cases using nobody is acceptable or even preferred, but in the specified case it is still an annoying and confusing bug. If you would like to say: this is not user friendly at all, and furthermore the documentation is not clear enough, misinterpretable (it is not mentioned anywhere: the cliend side settings or server side settings or some default assumptions have priority). My assumtion was defaults only used when required, but as I wrote already, rsyncd overkills client side settings. You tried to explain nobody is normal as default - and I accept it, I tried to explain overwriting/ignoring -perms is a bug... András -Original Message- From: Voelker, Bernhard [mailto:bernhard.voel...@siemens-enterprise.com] Sent: Friday, August 10, 2012 10:01 To: András Porjesz; Steven Levine; rsync@lists.samba.org Subject: RE: cannot rsync when source directory lacks write permission András Porjesz wrote (Friday, August 10, 2012 9:27 AM): I think all the permission/ownership handling is complicated (unaccountable, puzzling, peculiar) and the usage is confusing and annoying I think you didn't get Steven's point: [...] The default is uid -2, which is normally the user nobody. I.e. even if your rsyncd is running as root, it drops it's priviledges by default by becoming 'nobody'. That is a pre-cautious security issue which meets the requirements for most installations which just serve files read-only. When you want to push things there, then you might either rsync-via-ssh to that server as root (or sudo there), or set 'uid' to root in rsyncd.conf. Have a nice day, Berny -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Time rsYnc Machine (tym)
Paul Slootman wrote: On Thu 09 Aug 2012, Linda Walsh wrote: Anyway, thanks for the history update. I have a feeling rsync is afraid to use memory -- and really, it should try to use alot of memory to optimize transfers, I have had rsync fail after using up 8GB memory + 4GB swap, so I'm very happy it does its best to minimize memory usage. Well it should size itself for the resources available...in any case up or down... OK, and now on a 6*TB* HD :-) This is on one of my backup servers: /dev/sdc 39T 33T 5.9T 85% /backup you fit a 39T partition on a 6TB HD? something slipped there as on mine. -- my media partition was 6T, not 6G. I'm always losing units... Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Persistent, sporadic failure in make_bak_dir mkdir
Through several versions up to and including the current 3.0.9 I experience sporadic, repeating (but not predictably) errors of this form: rsync: make_bak_dir mkdir /REPO/backup/-data1-20120727-040453 failed: File exists (17) rsync: keep_backup failed: /REPO/backup/-data1/data1/A182/tb5178/tb5178.hebb/out.TENT_warn.txt - /REPO/backup/-data1-20120727-040453/data1/A182/tb5178/tb5178.hebb/out.TENT_warn.txt: No such file or directory (2) rsync: stat /REPO/backup/-data1-20120727-040453/data1/A182/tb5178/tb5178.hebb/out.TENT_warn.txt failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1518) [generator=3.0.9] About 20 of the last 2000 total nightly back up jobs have failed, almost all of them with this kind of error. They never reoccur when I rerun the job. The files/directories involved appear to be random. Is there anything I can do? Richard Crane --- Haskins Laboratories / (203) 865-6163 X 275 / FAX (203) 865-8963 300 George Street/New Haven, CT 06511 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: cannot rsync when source directory lacks write permission
In 64fab8215d47a944abdf7de50a3406a219c2dac...@esesscms0353.eemea.ericsson.se, on 08/10/12 at 09:27 AM, András Porjesz andras.porj...@ericsson.com said: Hi András, We may have to agree to disagree on some of this... Of course, the destination side must have enough rights to achieve what I need, and an rsyncd running as root:root surely have all the necessary rights. So using --perms with this daemon must have been sufficient. No, it's not the way it works and not the way I would want it to work. I do not want the server to automatically assume that the client can be trusted to do the right thing with data that is owned by the server. The fact that the rsync daemon is running as root is irrelevant. As you know from your many years of *ix experience, it is normal for a application running as root to use setuid to limit the rights it has. In my experience, it is typical for a server to do setuid nobody, unless instructed otherwise. But not, it is simply not true, because the daemon has a built-in assumption which overkills it. I know rsync sends its command line parameters to the daemon, so rsyncd could have been able to handle the case, but instead of recognizing the flag it simply makes that assumption. The assumption it makes is that the rsyncd.conf is correct when there is a choice to be made. This make absolutely good sense to me since in my world, the server knows what's best and the client is less trusted. 1. use exactly the same options on both sides This already happens. What does not happen is that the server does not automatically run as root. 2. specify which attributes to transfer (keep) and which ones to set explicitly (including permissions, ownership, time and probably acl also) This already happens. It is the client's responsibility to do this. The server will do it's best to fulfill the request. Failures can and will occur it the client asks the server to do something that exceeds its rights or is something the platform does not support. For example, I maintain an rsync port for a platform that does not support hard links. There's a whole set of client options that generate errors if the client requests them. 4. define a precedence (like: source filesystem, sender config, receiver config, receiver user rights, defaults) 5. describe actions taken in case of insufficient rights Most of this is already in place. If the server has insufficient rights the request will fail in some way and rsync will proceed, depending on options such as --ignore-errors. (just another example: client side -E, server side incoming chmod u+x,g-x and outgoing chmod u+x,g-x: what is the expected result when sending or receiving files?) As the man page says snip o To make a file executable, rsync turns on each x permission that has a corresponding r permission enabled. If --perms is enabled, this option is ignored. /snip Using aliases these can be mapped to the actual flags - for backward compatibility Huh? Steven -- -- Steven Levine stev...@earthlink.net eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Serious issue: rsync and hardlinks are dangerous...
Also, this is already reported, but a bit forgotten (last comment was three years ago): https://bugzilla.samba.org/show_bug.cgi?id=3693 I think if search the zilla, you find quite some open ticket on hardlink issues still? :( As in some of those tickets, provide your replicate command sequence and results to the tickets. Not forget 'ls -ali' too. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
