Re: Incremental file-list recursion has landed in CVS; Re: RSYNC + iNotify
Way back, On Fri, Jan 12, 2007 at 08:12:30AM +, Wayne Davison wrote: On Thu, Jan 11, 2007 at 05:22:55PM -0500, Matt McCutchen wrote: Specifically, I'm curious about what areas under the source argument(s) are scanned at what time. All the args that the user supplies are scanned at once, allowing them to be unduplicated as they would be in a normal transfer. The only difference is that no recursing happens during the initial sending of the first file-list. Then, one directory at a time is scanned and sent over until we have a decent number of upcoming files in the pipeline for the generator. The result is that typical transfers with a small number of source args can start transferring files almost immediately, and the depth-first scan of directories continues intermixed in with the file transfers (or at least intermixed with the generator's scanning for changed files when no transfers are needed). If the --relative option is used, implied directories are treated as args and sent in the first wave. The --files-from option has always treated the items read from the file as args, so a transfer with a huge number of files-from items and no real recursion doesn't get any benefit from the incremental recursion. Also, does the incremental scan rule out file has vanished warnings? It lessens their chance of occurring because the time that elapses between a directory scan and the time the generator starts to work on those files is much shorter than waiting for the full scan to complete. However, there can still be vanished files as there is still some reading ahead of directories (rsync tries to keep a good amount of work in the pipeline for the generator to blaze through). I haven't decided exactly what I want the read-ahead limit to be, but the current code wants 1000 files to be available beyond the currently-active directory. i may be reading the code incorrectly, but it seems that, if the --files-from option processing can be altered (or perhaps yet another option could be created [shudder]) to opt out of the de-duplicate pass and somebody hooked inotifywait http://inotify-tools.sourceforge.net/#info to the standard input of rsync -r --incremental-dir --files-from=- ... (and inotifywait can be convinced to fflush() after printing each event and somebody also took appropriate precautions to de-duplicate entries within a reasonable time frame etc. etc.) then you'd have sorta like what, even further back, On Wed, Feb 8, 2006 at 06:12:57PM +, Dag Wieers wrote: On Tue, 31 Jan 2006, Ryan Kather wrote: I'm looking for a way to continually monitor at least one but possibly multiple directories (and/or individual files). I would like RSYNC to immediately synchronize the changes to said directory(ies) after they occur. I believe the best approach for this would be to utilize iNotify enabled kernels and create a plugin for the RSYNC daemon. However, before I begin the task of actually writing some code (with my poor abilities), I thought I would inquire if anyone else has already created this or something similar? Am I over thinking this, or is there a better approach? Is there a reason not to do this? I'm very interested in functionality like this. I remember it being brought up on this list before so I would look for similar mails in the archive for clues. How to do it efficiently (eg. for files in transit/still open), I don't know. Also it seems to me that you may want a seperate daemon that implements the rsync protocol itself (instead of relaying on an external tool) as that allows you to optimize certain things and have less overhead. I'm most interested in writing this in python, using a python-rsync implementation and python-inotify. Kind regards, -- dag wieers, sorry if this is wacko or retraces ground already covered on the list (haven't been paying attention for a while, since rsync does everything i could possibly need--except, of course, just this 1 more thing, which i disclaim any liability for proposing, if the camel's back should break, security- or otherwise) or otherwise amounts to a waste of time, but the possibility of continuous rsync-ing, without you having to make but a few concessions in your code, seems like it might be worth making an idiot out of myself (a NOP) to suggest. (on Linux, i mean, though BSD must have similar kqueue/kevent tools available, i'd suppose) but if i just missed the news that somebody's found a good way to do it in the mean time, i'd be happy to hear about it. (google is keeping mum, if so) thanks again for all your work on this, and a last question: does any command have as many command-line options as rsync? are there as many atoms in the universe as combinations of rsync options? would adding a --sequential-files-from get you there? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before
Re: [Acl-Devel] mask ACL
On Tue, Apr 01, 2003 at 11:09:32PM -0500, Buck Huppmann wrote: yes, my bad. sorry. before i throw out more babies with the bathwater, though, anybody know if any other systems besides HP-UX and Solaris (for default ACLs, at least) require a MASK/CLASS_OBJ when there are no non-USER_OBJ/GROUP_OBJ/OTHER entries? thanks, Eric C., for finding this out to wind this up, for anybody who cares, the latest, greatest versions of this patch are up at http://www.lpmd.org/rsync/ (thanks to John C. again for hosting) for 2.5.5 and 2.5.6. use at your own risk, but let me know if you find bugs (so i can mitigate my own risk) on the matter of which platforms require masks/CLASS_OBJs, i gave the samba sysacls.c a quick once over and discerned that, at least for the mappings as implemented therein, HP-UX, Unixware and Solaris re- quire CLASS_OBJs, which the new code synthesizes by or-ing all group and named-user ACEs, as you might expect, in the absence of a source mask/CLASS_OBJ entry -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [Acl-Devel] mask ACL
yes, my bad. sorry. before i throw out more babies with the bathwater, though, anybody know if any other systems besides HP-UX and Solaris (for default ACLs, at least) require a MASK/CLASS_OBJ when there are no non-USER_OBJ/GROUP_OBJ/OTHER entries? thanks, Eric C., for finding this out -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Copying EAs and ACLs
On Sun, Feb 23, 2003 at 01:13:56PM +0100, Andreas Gruenbacher wrote: Hello, I am the guy behind the ext2/ext3 patches for Extended Attributes and ACLs, and I've recently been asked about ACL support in rsync by Eric. Upon investigating I found that you have an ACL patch against rsync-2.5.5 [1]. I also found some other postings to [EMAIL PROTECTED] concerning rsync and ACLs [3]. Are there any plans for finalizing an integrating that rsync ACL patch? i hope not seriously, though, the work i did was--we hoped--something to tide us over until something better came along. i have no illusions about or desire to see its incorporation into the official source tree i think that full-blown, general EA support is a laudable goal; my hope was just to preserve--to the naive extent possible by stealing a lot of other people's code and introducing as few of my own bugs as possible-- any additional permissions our folks are putting on their files so that our rsync-ed ``mirrors'' of their stuff aren't any less secure than their source images. in our case, simply preserving extended/discretionary ACLs between our Solaris (and, it's hoped some day soon now, Linux) systems was all we were looking for, and i'm content to live in ignorance of the more arcane^H^H^H general issue of EA support, since it's not anything that bears on security for our Solaris filesystems. (at least, i'm not aware of any other attributes that have significance security-wise; which is to say that i hope our users haven't figured out any yet) nevertheless, it seems to me also that the NFSv4 approach is a bit verbose and a bad enough fit for POSIX and Solaris/SysV ACLs, at least, that i would think twice about dumping my patch in favor of an implementation along those lines--at least if there weren't EAs that would benefit from a fully generalized, all singing, all dancing implementation. (but then again i am the sort who has trouble coping with the additional dimension- ality of access control beyond the POSIX triple, think NT permissions are a perplexing if exotic security nightmare, prefer single-letter options to --gnu-style options, prefer man to info, coffee to latte etc.). and yet i understand that somebody's going to figure out something cool to do with extended attributes that users are going to want to have rsync preserve for them and, in the future, it may be useless to transfer files at all with- out such things (just think, filesystem cookies and referral paths--gag). so maybe it's best to bite the bullet now, but i'm certainly not qualified to do it. then again, maybe you wait and see if NFSv4 and supporting file- systems force the issue, then you worry about it, just like seems to be the present rsync development strategy . . . (whence, bad patches like mine crop up to fill the void) sorry to ramble cynically, but thanks for showing an interest in clearing up the state of affairs and for your excellent work on ext2/ext3 EAs and ACLs and libattr and libacl -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: when using --backup-dir: why not make_bak_dir() . . .
sorry about the delay getting back to you on your considered reply,
sorry to see that this question has been asked before
(http://lists.samba.org/pipermail/rsync/2001-September/004877.html)
and sorry for this basically frivolous reply:
i guess my take is that, if the user requested preservation of per-
missions, then that may be what the user is expecting with respect
to backup-hierarchy recreation also. (i'll admit, for the record,
that *this* user was.) it could be sort of an unpleasant surprise,
at any rate, to find out that some file that one of your users hadn't
properly sured up permissions on but had been securely buried in a
hierarchy of drwx***--- directories had suddenly been made available
for public consumption by your backup regimen
so, far removed from any profiling considerations (which you can
correctly assume i'm unqualified to comment on), i guess what i'm
asking is that you Think About The Children--urrh, humble, unwit-
ting users like me, who would rather have butt-coverage than speed
in this regard and who wouldn't hold backup-futzing against rsync's
speed rep, especially if you gave us an option like
--slow,secure-backup-dir-permission-preservation
to warn us (and our weary typing fingers) away
thanks again, and wishing everyone also a Merry Christmas and Happy
New Year, if you observe those around this time, but Blessing and
Peace in any case
--buck
P.S.--for the record and on the subject of hobbling rsync, JMC
updated the ACL patch at http://www.lpmd.org/rsync-2.5.5-acl.diff.gz
to preserve ACLs on --backup -ed stuff. thanks, JMC, and caveat usor
On Mon, Dec 16, 2002 at 04:00:53PM -0600, James Delahanty wrote:
Buck,
You have a good point about make_bak_dir(), and pose a good question...
With the --backup-dir switch, any file *or directory* that is destined
for deletion is saved to the backup directory. A file is deleted by
rsync whenever it is actually deleted on the sync system (obviously!),
or when it is replaced by a file with the same name/different rsync
signature as part of the rsync process. All files or directories that
fall under this umbrella are saved to the backup-dir with their original
permissions intact.
The interesting question is: What do we do with directories that we must
manufacture to position the files in the right place in the
backup-dir?
My take was: Completely ignore the permissions of these directories,
unless they were deleted (see above), and use the robust_move()
function already in backup.c to move the file/create the directories as
needed (perms: 0755). This was done strictly for speed. The win: Fast,
no hassle path creation. The lose: Created path does NOT have same
permission as original path to file -- but the individual file
permissions are correct. This was an acceptable tradeoff for our
purposes, but YMMV. I can certainly understand (and appreciate) that
it's an unacceptable tradeoff to some.
Personally, I prefer that this shortcut be left as is, or be (yet
another) compile-time option. (-DKEEP_ALL_BK_DIR_PERMS), but I'm not a
maintainer, just a contributor. (You might like to profile the change,
though, just to see the kind of hit you take. IIRC, it was pretty
significant.)
P.S. -- Many thanks to the Rsync team, Merry Christmas and Happy
Holidays!
Regards,
Jim Delahanty
Date: Mon, 9 Dec 2002 17:13:15 -0500
From: Buck Huppmann [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: when using --backup-dir: why not make_bak_dir() . . .
here (in keep_backup())?
--- backup.c.orig Mon Dec 9 17:02:36 2002
+++ backup.cMon Dec 9 17:03:50 2002
@@ -273,6 +273,7 @@
/* move to keep tree if a file */
if(!kept) {
+ make_bak_dir(fname,backup_dir);
if (!robust_move (fname, keep_name))
rprintf(FERROR, keep_backup failed %s - %s : %s\n,
fname, keep_name, strerror(errno));
this reconstructs parent directory permissions more in keeping with
my (humble) expectations, in addition to being consistent with the
way all ~S_IFREG links' parents are produced
just curious
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
when using --backup-dir: why not make_bak_dir() . . .
here (in keep_backup())?
--- backup.c.orig Mon Dec 9 17:02:36 2002
+++ backup.cMon Dec 9 17:03:50 2002
@@ -273,6 +273,7 @@
/* move to keep tree if a file */
if(!kept) {
+ make_bak_dir(fname,backup_dir);
if (!robust_move (fname, keep_name))
rprintf(FERROR, keep_backup failed %s - %s : %s\n,
fname, keep_name, strerror(errno));
this reconstructs parent directory permissions more in keeping with
my (humble) expectations, in addition to being consistent with the
way all ~S_IFREG links' parents are produced
just curious
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
patch: rsync-2.5.5: UNIX ACL support
there's an unsupported, it-works-for-us patch that's seen a bit of testing on Solaris 2.6/8 and rudimentary testing on Linux with Andre- as G's ACL implementation. most of the code is ripped straight out of the samba 2.2.5 source tree; the rest of it is stuff that i don't expect anybody to ever seriously consider merging into the rsync tree and which i wouldn't defend against claims that it could wreck your machine, your career, your life, your mojo etc. http://www.lpmd.org/rsync-2.5.5-acl.diff.gz or ftp://ftp.lpmd.org/pub/rsync-2.5.5-acl.diff.gz (thanks, JMC, for hosting) to apply to a 2.5.5 tree, % cd wherever/rsync-2.5.5 % gunzip whereverelse/rsync-2.5.5-acl.diff.gz | patch -p0 \ ./configure --with-acl-support=yes ... \ make clean all to use, throw in a `-A' option. if it doesn't work, make sure both your local and remote (if different) rsync's have been patched: % rsync --version | grep acl Capabilities: 64-bit files, socketpairs, hard links, acls, symlinks, batchfiles, % ssh remote rsync --version \| grep acl Capabilities: 64-bit files, socketpairs, hard links, acls, symlinks, batchfiles, aside from that, the failure modes are largely unexplored i encourage anybody who has the hacking muse and the inclination to fix or proffer a better patch and maybe even to see about getting something merged into the project. i think, at any rate, purloining the samba guys' work is a decent basis for such an endeavor. anyway, my only commitment to ongoing maintenance is to update for 2.5.6, since that release seems imminent, and then whatever releases we keep in step with at my work -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
