Re: LGPL relicense port of rsync
> > > > > > > I guess I could write an initial protocol specification - but it would > > > not be complete and I wouldn't be able to relicense my library to > > > LGPL anyway. > > > > > > So I guess I have convinced myself that it is not worth the effort > > > trying. Time is probably better spent coding ;) And that's OK too, it > is not > > > that big of a deal anyway. > > > > Or think about following. You insist that your Java library is > > derivative work from the C program. OK. However, I believe a > > "translation into other languages" doesn't mean you make changes into > > the workflow by code restructuring, introducing another data > > structures, classes and so on. More such changes you made, less it just > > a "translation" and more an inspiration. Often I read in code not > > "based on" but "inspired by". > > > > Anyway, you have written every line in Java. This means you're a > > copyright holder on this. Thus you're allowed to license your work as > > you wish. In case you still insist it is a derivative work, you're > > required to allow the usage of your code under GPL. But! As a copyright > > holder you're allowed to give an arbitrary license additionally and > > even on a per case basis. > > > > This was my opinion. Additional references to approve or disapprove are > > welcome :) > > You might be right but I am a bit hesitant. > > > http://programmers.stackexchange.com/questions/58338/when-porting-code-must-i-follow-the-original-license > > http://programmers.stackexchange.com/questions/90232/original-author-rights-in-a-licensed-software-project?rq=1 > > http://programmers.stackexchange.com/questions/86754/is-it-possible-to-rewrite-every-line-of-an-open-source-project-in-a-slightly-dif These are talking about different situations: - 'porting' in the sense of making code run on a different platform while still having some code in common - line-by-line rewrite or translation - writing a new program using the rsync source as documentation of the protocol, as you are doing In my (not a lawyer) opinion, the last of them does not create a copyright derivative, and (separately) I don't object to you doing that on GPL'd work that I wrote. I would consider the first two to be a violation. I think you have a couple of cheap options to get some clarity: - mail the other key authors listed above explaining what you're doing and ask if they object - mail the FSF or SFLC as custodians of the L/GPL > I think that the best thing would be if rsync would be split into a > library part (LGPL) and application part (GPL). This could make the > rsync protocol even more used. > > But again, it could be quite some substantial work, both coding (?) > but also getting permissions from previous contributors to relicense > the library part. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Fwd: Delete some excluded files in rsync
Begin forwarded message: From: Karel Kulhavy <[EMAIL PROTECTED]> Date: 7 March 2006 18:01:43 To: Martin Pool <[EMAIL PROTECTED]> Subject: Delete some excluded files in rsync Hello I suggest that a feature be added into rsync. That one could separately specify excluded files that should be deleted on the receiver and excluded files that shouldn't be deleted on the receiver. I am using rsync for remote updating of my website http://ronja.twibright.com and this feature would be handy because some files are generated on the server because cannot be generated on the laptop where the files are edited, and shouldn't be deleted. The remaining excluded files should be deleted on the receiver if they got accidentally copied in the past, for example becaue the rsync script wasn't tuned properly. Regards, CL< -- Martin Pool -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [librsync-users] MD4 second-preimage attack
On Tue, 2006-02-21 at 14:58 -0800, [EMAIL PROTECTED] wrote: > A year ago we discussed the strength of the MD4 hash used by rsync and > librsync, and one of the points mentioned was that only collision > attacks are known on MD4. Could you please forward this into the bug tracker so it's not lost? -- Martin signature.asc Description: This is a digitally signed message part -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Spam to this list
John Van Essen wrote: Off list to rsync list owner (feel free to reply on-list if you like): On Fri, 25 Mar 2005, Dag Wieers <[EMAIL PROTECTED]> wrote: Hi, I'm not sure what the policy of this list is and I bet everyone has a spam filter, so nobody might have noticed, but we got spammed. The policy is to block as much spam as possible without blocking legitimate posts. A 100% solution is impossible, even if we had human moderation (humans make mistakes). It seems that these posts got through during a surge of spam when the filter hit its maximum-process limit. During the day of the 24th more than 60 spam messages to the list were blocked. I got several. Delivered to the mailing list from: cpe-24-243-54-175.satx.res.rr.com [24.243.54.175] unknown [219.252.105.93] unknown [218.59.89.16] unknown [200.159.206.55] The first one has been in the dul.dnsbl.sorbs.net blacklist since Oct. I use these 4 DNS-based blacklists in the mail server that I manage: sbl-xbl.spamhaus.org list.dsbl.org dul.dnsbl.sorbs.net web.dnsbl.sorbs.net And they have helped a LOT. The other 3 have no reverse DNS entries. A machine with no reverse DNS that is sending email is not very likely to be a legitimate email server. It's much more likely a compromised machine on a clueless ISP's network. Rejecting email from those unidentified machines also has helped a lot. Using any of those measures alone tends to block legitimate posters, particularly those running their own mail server, which to my mind is a greater harm than letting ocassional spam go through. Our purpose here is to run a mailing list, not punish ISPs. So we use all the things you named as part of a weighted score. -- Martin signature.asc Description: OpenPGP digital signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync filename heuristics
On 5 Jan 2005, Rusty Russell <[EMAIL PROTECTED]> wrote:
> On Tue, 2005-01-04 at 18:24 +0100, Robert Lemmen wrote:
> > hi rusty,
> >
> > i read on some webpage about rsync and debian that you wrote a patch to
> > rsync that let's it uses heuristics when deciding which local file to
> > use. could you tell me whether this is planned to be included in a rsync
> > release? could i have that patch?
>
> Hmm, good question. This is from 2.5.4, and can't remember how well it
> worked. Good luck!
I'm not the rsync maintainer anymore, but I think it would be cool if
this were merged, if the current team feels OK about it.
>
> Rusty.
>
> diff -urN rsync-2.5.4/Makefile.in rsync-2.5.4-fuzzy/Makefile.in
> --- rsync-2.5.4/Makefile.in 2002-02-26 05:48:25.0 +1100
> +++ rsync-2.5.4-fuzzy/Makefile.in 2002-04-03 16:35:55.0 +1000
> @@ -28,7 +28,7 @@
> ZLIBOBJ=zlib/deflate.o zlib/infblock.o zlib/infcodes.o zlib/inffast.o \
> zlib/inflate.o zlib/inftrees.o zlib/infutil.o zlib/trees.o \
> zlib/zutil.o zlib/adler32.o
> -OBJS1=rsync.o generator.o receiver.o cleanup.o sender.o exclude.o util.o
> main.o checksum.o match.o syscall.o log.o backup.o
> +OBJS1=rsync.o generator.o receiver.o cleanup.o sender.o exclude.o util.o
> main.o checksum.o match.o syscall.o log.o backup.o alternate.o
> OBJS2=options.o flist.o io.o compat.o hlink.o token.o uidlist.o socket.o
> fileio.o batch.o \
> clientname.o
> DAEMON_OBJ = params.o loadparm.o clientserver.o access.o connection.o
> authenticate.o
> diff -urN rsync-2.5.4/alternate.c rsync-2.5.4-fuzzy/alternate.c
> --- rsync-2.5.4/alternate.c 1970-01-01 10:00:00.0 +1000
> +++ rsync-2.5.4-fuzzy/alternate.c 2002-04-03 17:04:15.0 +1000
> @@ -0,0 +1,117 @@
> +#include "rsync.h"
> +
> +extern char *compare_dest;
> +extern int verbose;
> +
> +/* Alternate methods for opening files, if local doesn't exist */
> +/* Sanity check that we are about to open regular file */
> +int do_open_regular(char *fname)
> +{
> + STRUCT_STAT st;
> +
> + if (do_stat(fname, &st) == 0 && S_ISREG(st.st_mode))
> + return do_open(fname, O_RDONLY, 0);
> +
> + return -1;
> +}
> +
> +static void split_names(char *fname, char **dirname, char **basename)
> +{
> + char *slash;
> +
> + slash = strrchr(fname, '/');
> + if (slash) {
> + *dirname = fname;
> + *slash = '\0';
> + *basename = slash+1;
> + } else {
> + *basename = fname;
> + *dirname = ".";
> + }
> +}
> +
> +static unsigned int measure_name(const char *name,
> + const char *basename,
> + const char *ext)
> +{
> + int namelen = strlen(name);
> + int extlen = strlen(ext);
> + unsigned int score = 0;
> +
> + /* Extensions must match */
> + if (namelen <= extlen || strcmp(name+namelen-extlen, ext) != 0)
> + return 0;
> +
> + /* Now score depends on similarity of prefix */
> + for (; *name==*basename && *name; name++, basename++)
> + score++;
> + return score;
> +}
> +
> +int open_alternate_base_fuzzy(const char *fname)
> +{
> + DIR *d;
> + struct dirent *di;
> + char *basename, *dirname;
> + char mangled_name[MAXPATHLEN];
> + char bestname[MAXPATHLEN];
> + unsigned int bestscore = 0;
> + const char *ext;
> +
> + /* FIXME: can we assume fname fits here? */
> + strcpy(mangled_name, fname);
> +
> + split_names(mangled_name, &dirname, &basename);
> + d = opendir(dirname);
> + if (!d) {
> + rprintf(FERROR,"recv_generator opendir(%s): %s\n",
> + dirname,strerror(errno));
> + return -1;
> + }
> +
> + /* Get final extension, eg. .gz; never full basename though. */
> + ext = strrchr(basename + 1, '.');
> + if (!ext)
> + ext = basename + strlen(basename); /* ext = "" */
> +
> + while ((di = readdir(d)) != NULL) {
> + const char *dname = d_name(di);
> + unsigned int score;
> +
> + if (strcmp(dname,".")==0 ||
> + strcmp(dname,"..")==0)
> + continue;
> +
> + score = measure_name(dname, basename, ext);
> + if (verbose > 4)
> + rprintf(FINFO,"fuzzy score for %s = %u\n",
> + dname, score);
> + if (score > bestscore) {
> + strcpy(bestname, dname);
> + bestscore = score;
> + }
> + }
> + closedir(d);
> +
> + /* Found a candidate. */
> + if (bestscore != 0) {
> + char fuzzyname[MAXPATHLEN];
> +
> + snprintf(fuzzyname,MAXPATHLEN,"%s/%s", dirname, bestname);
> + if (verbose > 2)
> + rprintf(FINFO,"fuzzy match %s->%s\n",
> + fname, fuzzyname);
> + return do_open_r
Re: A question about rsync
On 7 Jun 2004, Guo jing <[EMAIL PROTECTED]> wrote: > Thanks for your answer! > Yes,my question is that if we can get a good result when the file is > changing while it is being copied by rsync > > In my test, if the file is being augmented while it been copied using > rsync.I can get a normal copy on the other end and the result file is the > same as what the source file is when the rsync scanning. The same result > can be gotten if the sour file is reduced and the blocks were not occupied. > > As you said, if the source file reduced and the blocks were occupied by > other files there will be a file with other file's content and a abnormal > end on the other end. > > So,is this true that we can't deal with this problem except to do some > changes with the OS ? Yes, or don't change the file while it is being copied. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: A question about rsync
On 31 May 2004, Guo jing <[EMAIL PROTECTED]> wrote: > hello, > I am a student in China.I like the linux and usually use the rsync to > backup my documents. Last week when I use it,I find a question I want to > discuss with you. > > The condition is like this: The source file that I want to rsync to > another computer is 129M before I start the rsync. During the running of > the rsync,the file was changed and became to about 50M, then the rsync > ended. When I view the destination, I found that the file was 129M. And > there were some contents of the files added when the rsync was running. > > After that, I do some tests about the rsync: > > 1. After I start the rsync to backup a file, I delete the file > during the rsync is running, I found the file can been backuped normally. > 2. While the rsync is backuping a file name sourfile (50M), I add > some content by the command "cat addfile >> sourfile" to enlarge the file > to 100M. After the rsync finished.I found the file is still 50M. > >The question is that , how the rsync copy a file to another computer at > the first time ? My attitude is that it remenbers the physical blocks the > file used when the rsync scaned. Then ,rsync will send the blocks to the > destination no matter if the file or the block has changed. So, is that > right?? Who can tell me how the rsync decide which contents should to send > to the destiation? I'm not sure I understand the question, sorry. If you change a file while it is being copied by rsync you may end up with undefined results on the other end. There is not much that can be done about this without os-level version control. > > Sorry, my English is very poor. Thanks for your read and answer!! > > _ > MSN Messenger: http://messenger.msn.com/cn > -- Martin signature.asc Description: Digital signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: I20 Drivers Crash system when used with Rsync
On 30 May 2004, "Dennis R. Gesker" <[EMAIL PROTECTED]> wrote: > Note: I don't know if this is a problem withe I20 drivers or Rsync so > I'm submitting to both the Kernel Bugzilla and the Rsync mailing list. I > couldn't find a bugzilla for Rsync. I hope this was the correct way to > submit this issue. > > Distribution: Debian > Hardware Environment: Intel 850MV Mother board, Pentium 4 processor, > 1Gig of RAM, Adaptec 2400A RAID Controler. Both the motherboard and > Controller card have the most recent BIOS/firmware installed. The > Adaptec card is capable of RAID configuration but currently it is > configured to view each of the attached IDE drives as individual drives. > None of the cards RAID features are presently beeing used. Network is > 100MB/s Switched Ethernet. Network cables and connects have been tested > and verified. > > Software Environment: Very basic/vanilla Debian system install (Sid > branch). Software package is rsync. > > Problem Description: When transfering many sometimes large files (>3Gig > in some cases)for backup purposes using rsync either via an ssh shell or > rsync server the I20 drivers cause a kernel panic. The system seems to > report increases in queue depth, shortly afterward the system completely > hangs indicating a kernel panic. A kernel panic is by definition a kernel bug, not an application bug. Good luck! :-) p.s. kernel bug reports ought to say what kernel you're using. > > Steps to reproduce: Transfer files using rsync. Last specific command > issued at prompt that reproduced this error was: > "rsync --bwlimit=2048 -vv -r -e ssh --delete --exclude lost+found > rsync://[EMAIL PROTECTED]:873:/bu/area1/blue/* /bu/area1/blue > " > > This error does not seem to occour when transferrring the same file set > using cp over nfs or scp. However, this does happen using rsync over nfs. -- Martin signature.asc Description: Digital signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug reporting
On 1 Jun 2004, John Summerfield <[EMAIL PROTECTED]> wrote: > The jitterbug link on http://rsync.samba.org/nobugs.html no longer works. I > suggest it either be fixed or removed. Thanks, fixed. > You make bug-reporting needlessly difficult, I think. I dislike the need to > subscribe to a mailing-list and potentially receive lots of email that > doesn't interest me. I have plenty of other email to keep me amused. I don't think you need to subscribe to post. I put the address directly on the nobugs page to make it easier for people to write to it. Did you have any other suggestions about how to make it better? The reason we took Jitterbug and faq-o-matic down is that people seemed to get help more promptly when they wrote to the list. > What I wanted most to do is ensure you know about rsyncx and consider working > with the authors to create a unified product that supports resource forks > when built on OSX. > > See http://www.macosxlabs.org./rsyncx/rsyncx.html > > Their CVS repository is at http://www.opendarwin.org > > It seems a shame to have two projects where one will do. Well, sometimes there are reasons not to glom everything into one big program. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
faq-o-matic gone
The rsync faq-o-matic was broken during the recent machine migration. Since there was relatively little useful content and a lot of unanswered or pointless questions, I am going to remove the links to it. If anyone feels like maintaining an FAQ please do so. -- Martin signature.asc Description: Digital signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
(fwd from psdasilva@esoterica.pt) rsync: Request for a feature
- Forwarded message from Paulo da Silva <[EMAIL PROTECTED]> - From: Paulo da Silva <[EMAIL PROTECTED]> Subject: rsync: Request for a feature Date: Sun, 02 May 2004 17:09:11 +0100 To: [EMAIL PROTECTED] User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040317 X-Spam-Status: No, hits=-0.9 required=3.2 tests=BAYES_10 autolearn=ham version=2.63 Hi. 1st. of all thank you for mantaining this very useful program. It helps me a lot in a lot of tasks that would be otherwise very tedious and time consuming. However, I think that a small feature could make it yet more powerfull when used as a backup tool. The idea was to have a switch so that files could be kept compressed at the destination. These compressed files could be then restored the same way specifing a switch telling that source files are to be uncompressed. Files with known extensions (.gz, .zip, ...) should not be compressed/uncompressed. All files must keep the original names unchanged. Ex. export RSYNC_COMPRESSED_EXTS=".gz .zip ..." ;# Extensions from files not to be compressed Backup: rsync -av --delete --zip MyDir/ BackupDir Restore: rsync -av --delete --unzip BackupDir/ MyDir This is only a sugestion. You may find a better solution. Thank you. Paulo da Silva - End forwarded message - -- Martin signature.asc Description: Digital signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: test message only
On 23 Apr 2004, Jim Salter <[EMAIL PROTECTED]> wrote: > This is a test message - my apologies for it, but everything I send is > getting bounced. Our spamfilter was a little too hasty. It should be OK now. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
OT: fyi, spam
Just as background information: our spam filter caught 14000 attempted spams in the last two weeks. Suggestions on blocking more are welcome but the vast majority is already blocked. I think we removed the @samba.org whitelist. -- Martin signature.asc Description: Digital signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync / ssh -i
On 4 Dec 2003, Michael <[EMAIL PROTECTED]> wrote: > I know that with ssh I can issue the -i command to use a different identity. > Is there anyway to use the -i command with rsync and ssh? Thank > you. Use the IdentityFile and Host keywords in your ssh_config: Host suzy-alt-key HostName suzy.foo.org IdentityFile ~/.ssh/id_some_other_dsa -- Martin linux.conf.au -- Adelaide, January 2004 signature.asc Description: Digital signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync-bugs and unclear semantics when copying multiple source-dirs to one target
On 24 Nov 2003, Dirk Pape <[EMAIL PROTECTED]> wrote:
> Dear Martin Pool,
>
> I tried to ask via the rsync-mailing list but never got an answer. So I
> contact you directly.
>
> I refer to the rsync syntax
>
> rsync [OPTION]... SRC [SRC]... DEST
>
> with more than one SRC, which is mentioned in the man-pages.
> We use this form to "overlay" a target directory tree from more than one
> sources (class, group1, group2, ..., machine) to yield a costomized
> "cloned" directory.
>
> There are some glitches and bugs when using this form of rsync commands,
> one of which I have described in the here attached mail to the rsync
> mailing list. This is a platform specific bug.
The heart of the problem is that you are trying to write the same file
from several different source directories. I think this just will not
work predictably in the current design of rsync, because it builds a
single list of all files at the start of the transfer. Furthermore
the order in which files are transferred is rather strange, for
reasons of historical compatibility.
I think we do not make any guarantees about what happens if the same
relative path occurs in several source directories the behaviour is
undefined.
I agree that it would be nice if it processed the source directories
in the order they are given, but that is not how it works.
At the moment your options are:
Fix rsync to support this behaviour.
Transfer the directories one at a time to build up the destination.
This has several problems, one being that there may be many
redundant transfers and another that the state will be inconsistent
for longer.
Make a single source directory that has the state you want.
Ditto, but use union bind mounts to synthesize it from several
directories, assuming that your OS supports that.
Use some other tool.
Do several rsync transfers using exclude/include options to pick the
right directories from each overlay.
The last is possibly the most promising. You could even write a
little Perl script to build the exactly correct include lists.
> There is another glitch, which I will describe here:
>
> if you have the following directory structure (-> is softlink)
>
> ./dir1/dir/a
> ./dir2/dir -> ../dir3/dir
> ./dir3/dir/b
>
> and do
>
> rsync -av --delete dir1/ dir2/ target
>
> you get
>
> ./target/dir -> ../dir3/dir
> ./dir3/dir/a
> ./dir3/dir/b
>
> I would expect either
>
> Variant 1:
> ./target/dir -> ../dir3/dir
> ./dir3/dir/b
>
> (contents of /dir1/dir is ignored because dir ist "overlayed" with a
> symlink in dir2)
> or
>
> Variant 2:
> ./target/dir -> ../dir3/dir
> ./dir3/dir/b
>
> (./dir1/dir/a is copied following the overlayed symlink *but* the --delete
> then also has to follow the symlink)
>
> I would prefer strongly to see variant 1 or a new option to protect target
> directories from changing contents by linking in o them.
>
> For your motivation:
>
> Our more complex scenario is like that: We have
>
> class/usr/share/bugzilla/
> machine/usr/share/bugzilla -> /local/usr/share/bugzilla
>
> and we do something like
>
> rsync -av --delete --exclude local class/ machine/ targethost:/
>
> the "--exclude local" protects files in targethost:/local from being
> deleted but not from being overwritten with files which are present in
> class/usr/share/bugzilla/ on the scr-host.
>
> I would like to see an option (or standard semantics) to simply "killing" a
> directories "sub"-filelist when the directory is overlayed by a symlink in
> a source directory given later in the command line. May be it would suffice
> to do that only if the symlink points to a directory, which is "outside"
> all source dirs or element of an exclude list.
>
> I hope you understand and can help me.
>
> Thanks,
> Dirk Pape.
> From: Dirk Pape <[EMAIL PROTECTED]>
> Subject: bug (filelist) for platforms solaris and darwin (macosx) and *not*
> linuxi386
> Date: Sun, 28 Sep 2003 13:19:45 +0200
> To: [EMAIL PROTECTED]
> X-Mailer: Mulberry/3.1.0b7 (Mac OS X)
>
> I have found a nasty bug when a file, which is in some of many sources,
> shall be copied to a target.
>
> The linux-Version works well but rsync 2.5.{2|5|6} under solaris9 (gcc
> 2.95.3) and darwin (gcc 3.1) do not. The decision which file (out of which
> src) shall be copied depends on the number of src dirs given on the command
> line.
>
> This bug bytes us very hard, because we decided to rely on rsync to build
> local directories by "overlaying" different directories from a server and
> need to be sure to have a consistent semantics in wha
Re: rsync & rcp
On 30 Oct 2003, [EMAIL PROTECTED] wrote: > > I was hoping that since you guys are the authors to rsync that > you could answer a simple question for me. > > I'm trying to transfer files via the rsh/rexec protocol by > remotely executing a cat command, i.e. "cat > foo.txt" > and then sending data through the socket to the stdin of the remote > process. This all works fine, except for the fact that I have to > close the socket to force and end of file. > > My question is, does rcp/rsync close a socket when it sends files > to signify and end of file? If not, how does it send multiple files > without closing the socket? It uses a binary protocol to delimit files and describe metadata such as their name and ownership. As you say you cannot use the end-of-file mark more than once. It is conceptually similar to a tar file. So if you wanted to send multiple files with just rsh, you could do tar c mydir | ssh somehost tar x [EMAIL PROTECTED] is a better forum for questions. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Filesystem panic
On 22 Oct 2003, Morten <[EMAIL PROTECTED]> wrote: > > Hi, > > I'm running RH9, 2.4.20-18.9. Each night, the server mounts > an external FAT32 disk using firewire, and performs backups > to it using rsync. > > Twice within the past 3 months, the backup process has resulted > in machine crash (complete hang, hardware reboot needed). > > From /var/log/messages: > > Oct 22 04:02:20 yoda kernel: Filesystem panic (dev 08:21). > Oct 22 04:02:20 yoda kernel: fat_free: deleting beyond EOF > Oct 22 04:02:20 yoda kernel: File system has been set read-only You probably need to report this to the vfat fs maintainer VFAT FILESYSTEM: P: Gordon Chaffee M: [EMAIL PROTECTED] L: [EMAIL PROTECTED] W: http://bmrc.berkeley.edu/people/chaffee S: Maintained > From the rsync error log, I can see that the filesystem becomes > read-only, and that it begins to fail the synchronization task with > > mkstemp > SAGER_igangvaerende/2003_07_Fremtidens_Uni/Fase_4/Diagrammer/Celletyper/ > .Delecelle.psd.Rz49bM failed: Read-only file system > > Which is understandable. After doing this for a while, the error > message changes to "Too many open files". And I suspect that this is > what causes the machine to hang. Can you please try to reproduce the problem, and then do lsof -p PID_OF_RSYNC for each rsync process sometime before it starts complaining about too many open files. Then kill rsync to avoid the problem. > Is there any way to configure rsync to abort execution once the > first error occurs? Not at the moment. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: The rsync daemon as a gateway service?
That's an interesting idea. As a temporary measure you might different tcp ports rather than module names to distinguish different services, and then use tcp redirectors. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: lock out of machine
[EMAIL PROTECTED] is a better forum] On 30 Sep 2003, [EMAIL PROTECTED] wrote: > > On 29 Sep 2003, [EMAIL PROTECTED] wrote: > > > Please help... > > > > > > I lock myself out of one of my server... I was testing > > > the rsync -p -e /etc/passwd from my server to another > > > server... > > > and forget to login into this other server before > > > executing the above command... > > > > > > the other is a linux server can i reverse the command on > > > the rsync server to get to /etc/passwd.org to the > > > /etc/passwd file > > > > I assume from your message that you're not able to log in > > using either the new or the old message on the other > > server. You should try that first. > > > > If your password file is broken so that you cannot > > remotely log in, then you will need to break in to the > > machine. > > > > The easiest way is to go to the console and reboot from a > > recovery disk. If you don't already have one, you can get > > one from lnx-bbc.org. Once you have control of the > > machine as root, you can restore or recreate the password > > file. > > > > Good luck! Let me know if you're still stuck. > > > Thanks > > Question- > I have a solaris 9 sunray server... I install rsync-2.5.6 on > the solaris an older version is on the linux (crossover > server) box I want to upload my solaris /etc/passwd, > /etc/shadow, and /etc/group file s to teh linux box.. when i > rsync -p -e ssh /etc/shadow client:/etc/shadow it works..and > whent i rsync -p -e ssh /etc/passwd client:/et/passwd it > works but when i rsync -p -e ssh /etc/group > client:/etc/group it does not work--i get Connect closed by > 0.0.0.0 > rsync: connection unexpectedly closed (0 bytes read so fat) > rsynce error: error in rsync protocol data stream (code 12) > at io.c(165) Are those files compatible between Linux and Solaris? I suppose they are if it's working so far. Perhaps try repeating the command with the -vv option to get some debugging output. > can you help...??? > > I want to put this in my crontabe file to update every 5 > minute..Do i make a /etc/rsyncd.conf file and if so what > should be the informat. in it...??? Working over SSH should be fine. In this case you don't need a rsyncd.conf file. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Logging updated files?
Why did you send six copies of this? -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync silently changes special files to regular ones!
On 25 Aug 2003 Clemens Fischer <[EMAIL PROTECTED]> wrote: > * jw schultz: > > > On Mon, Aug 25, 2003 at 05:22:39PM +1000, Martin Pool wrote: > >> On Sun, 24 Aug 2003 16:44:15 +0200 > >> clemens fischer <[EMAIL PROTECTED]> wrote: > >> > rsync should error exit or message the user when used on special > >> > files! > > >> No, I don't think so. At the most I could stretch to giving a > >> warning when replacing a special file, but even then I'm not sure. > >> Unix is "you asked for it, you got it." > > this is changing a bit over time. i've been into unix for the reason > you stated and "if you ask something not available, script it up!". > yet everybody toys around with new things, and rsync/rdiff had not > sunken in by the time i did that "rsync ... /dev/stdout". maybe i > thought: "hey, it's a smart copy command, lemme see that file." You said "copy the file", so it copied the file. Oh, you wanted *the contents of* the file? You should have said so. > cope with device files how? RTFM please. It can copy device nodes as device nodes, when run by root and given the --devices or --archive options. This is the same as GNU cp. It should not be surprising. > and when this "is well within its > expected function", how in the world could i ever expect wrong? as > you can't possibly anticipate what people "expect", wouldn't it at > least be nice to include a warning in the man page? Can you suggest a good phrasing? > i had read both rsync(1) and rsyncd.conf(5) several times, as can be > seen by my ability to set up a working rsync server, but i did make > that error. this one caused a couple of really superfluous messages > to a local mailinglist and some to freebsd-stable, because breaking of > the CVS installation procedure(!!) was the first sign of /dev/stdout > beeing out of commission that appeared on my box. had this been a > production server ... I'm sure it was an unpleasant experience. However, you could have caused the same damage with rm, cp, or mv, so I'm not sure why you're upset with rsync... > all i ask is a short sentence like "think twice before rsyncing to a > local special file like /dev/stdout at.al., it may well do something > you didn't want to happen!". Nothing personal, but that is an absolutely terrible sentence for a manual: it creates worry and fear without providing any useful information. Perhaps something like Be aware that the destination files are replaced with a new node, not updated in place. > besides, what could be the sense in rsyncing to /dev/ as > destination? does rsync handle raw magnetic tape, ZIP drives etc.? > rsync would at least need a device capable of [l]seek(2)s, right? Surely this is in the faq, isn't it? -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Operation not permitted?
On 9 Sep 2003 "Max Kipness" <[EMAIL PROTECTED]> wrote: > Can someone tell me what the problem is here. I am doing an rsync on a > sendmail spool directory to a folder that is a samba mount. What do you mean by "a samba mount"? A filesystem mounted over smbfs? > Why is rsync trying to change owner? Because you told it to, using the -a, -o or -g options. > Does it have to? You asked for it, you got it :-) > I tried manually changing owner (as root) on a file that is sitting on > the samba mount and I got the same operation not permitted error. Assuming you're using smbfs, it's because smbmount logs in to the server as a single NT user, and all files appear to be owned by that user. Ownership is not preserved. cifsfs may fix this, but you need to ask about that elsewhere. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Looking for atime reset...
On 9 Sep 2003 "Saylor, Ted" <[EMAIL PROTECTED]> wrote: > I find rsync an excellent tool when I need to move multi-gigabyte > filesystems, because I can do most of the copying during the week - > then a quick cleanup sweep in our 4 hour outage window. > > I do need to somehow get the atime's to copy over, because as it > stands now I loose the age information (which we will soon be using > for auto-archiving) on things I copy with rsync. > > Would it be that hard to enhance rsync to copy the atime along with > the current mtime info? > > Does anyone have a speedy script, perl, or C program to "cleanup" the > atime after the final rsync is done? You don't say what operating system or filesystem you're using, but on Linux there is no standard way to change the atimes of a file, so there is nothing rsync can do about it. If you persuade your friendly neighbourhood kernel hacker / vendor to add an operation to do this then I suppose rsync could support it. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsyncing *to* live system
On 26 Aug 2003 jw schultz <[EMAIL PROTECTED]> wrote: > On Wed, Aug 27, 2003 at 09:25:41AM +1200, Steve Wray wrote: > > Hi there, > > I have been asked to develop a system for keeping > > a bunch of machines remotely configured and updated. > > The client has asked for this to be implemented using rsync. > > > > The machines involved are located at remote sites and > > physical access by competent personel is non trivial. > > And the systems are running Debian. > > > > I am a little concerned at the prospect of using rsync to > > replace running binaries, open files and system libraries. > > I've searched for an example where rsync has been used in this way. > > > > So far I have found nothing; people use it to backup a live > > filesystem; we are tasked with doing the reverse (sort of). > > And there are people who use rsync to replicate systems (rolling out > > a bunch of identical boxes; typically these recieve the rsync > > *before* they go live not after). > > > > So, can anyone please give me arguments or reasons for > > or against using rsync in this way? References to sites > > which currently use rsync in this way would be much appreciated. > > There are some difficulties that can occur depending on how > you structure your filesystems. > > It is possible to produce temporary dangling symlinks. > Rsync may remove the destination of the link before > the symlink is updated or deleted (see --delete-after); or > if rsync creates or updates a symlink before the destination > is created. > > You can get inter-file inconsistencies. The file sets are not > updated atomically so different config files and binaries > may be updated at slightly different times. Because rsync > processes the file list in lexical order the window size will > depend on the relative remoteness of files in the directory > hierarchy so files in the same directory have small windows > but files in different subtrees will have a somewhat larger > window. Here is an example of a bad case: a program depends on a shared library, and needs to be recompiled when a new version of the library is released. Your transfer upgrades the program before it updates the library (or vice versa) and the program crashes. I agree with JW and will just add that the inter-file inconsistencies could be far worse if the transfer is ever interrupted due to e.g. a network outage. If you interrupt it at the right (or wrong) time it is possible that rsync will no longer be able to run. dpkg knows how to upgrade software in a safe and sane way, avoiding all these problems. Let it do its job. By all means use rsync to transfer the packages, but then run apt or dpkg. In addition, once you upgrade software, you will want to restart daemons to make sure the upgraded stuff is used. dpkg handles that too... -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: performance suggestion: sparse files
On 9 Sep 2003 "Jon Howell" <[EMAIL PROTECTED]> wrote: > > Actually you can guess by looking at the allocated-blocks measure, > > and use this to guess whether it's preallocated zeros or sparse, > > which might be useful for backups. But there is no way around > > reading the blocks. > Sure. Bummer; that's a lot of memory bus bandwidth (having the kernel > zero-fill the blocks, then having rsync zero-compare them) wasted. If the program mmaps the file the kernel will fill the vm with COW references to the zero page and it will be quite cheap. > Seems like a fcntl() is in order. Repeat after me: premature optimization is the root of all evil. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: doing an md5sum rsync?
On 9 Sep 2003 Greger Cronquist <[EMAIL PROTECTED]> wrote: > See also unison, http://www.cis.upenn.edu/~bcpierce/unison/ which does > exactly this (and synchronizes using the rsync algorithm). Yes, Unison is very cool. I hadn't realized that it detected renames though. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: performance suggestion: sparse files
On 26 Aug 2003 jw schultz <[EMAIL PROTECTED]> wrote: > On Tue, Aug 26, 2003 at 11:28:12AM -0700, Jon Howell wrote: > > I worked around the problem by adding -z to compress the stream > > first(blocks of zeros compress remarkably well), and that made the > > virtual disk image transfer go much faster. Of course, all of the > > .tgzs and .tbzs in the same transfer got slower waiting on the > > source CPU to compress the incompressible. > > That is what i would have recommended. > > > The obvious solution is to change > > the protocol, but that seems like a scary thing to do for a > > performance tweak. What about an option for > > "really-crappy-compression"? Something really cheezy (RLE) that can > > decide in a hurry whether to compress away a string of zeros, and if > > not, just send them raw. That way, performance on compressed files > > stays I/O bound even on systems with pokey CPUs, but sparse files > > are disk-bound on the source system (as they should be). (And, of > > course, --sparse would automatically promote the compression level > > to "really-crappy" if it was at "none" before.) > > This is really only an issue when rsync hits a new file. I > agree an RLE of the stream _sounds_ lika a good idea. But > even better might be an extra phantom block that represents > all zeros. That too would require a protocol bump. I'd want to be convinced that this was really enough cheaper than -z1 to justify the complexity. (For rdiff having cheap encoding of zeros would seem to make sense...) > There is no way in user-mode to distinguish between a sparse file and > a file full of zeroed blocks. That is correct. Actually you can guess by looking at the allocated-blocks measure, and use this to guess whether it's preallocated zeros or sparse, which might be useful for backups. But there is no way around reading the blocks. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Add a feature : disk and partition cloning
On 2 Sep 2003 "francis.mit" <[EMAIL PROTECTED]> wrote: > Bonjour, > > Today, I use rsync for updating some 40 Debian/Linux box, rsync is > great. > > So, now, I'll need to update a whole disk or partition (NTFS) with an > image or an other disk or part. (case multiboot system), > can'I hope rsync do this task in some day ? I agree that it would be a cool feature. It's unlikely that the existing codebase would be extended for it, but something like rdiff might support it eventually. In the meantime, just dd across ssh. > rsync algorithm would be great for this task, isn't-it ? Not directly; the basic rsync algorithm cannot update in place. You might adapt it to do so though. > I'don't mind how amount of works this feature need, > but some folks are interresting in ? You don't mind how much work other people do for you? How gracious. Or were you volunteering to write it? If so, adding in-place updates to rdiff would be a good place to start. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: doing an md5sum rsync?
On 7 Sep 2003 Marc MERLIN <[EMAIL PROTECTED]> wrote: > I don't know if this has been requested before, but I would really > like for rsync to compute an md5sum for each file at the source and > destination (with a flag turned off by default of course), and it > would realize that I renamed files at the source by noticing a > matching md5sum between different filenames > It would then rename the destination instead of deleting it and > resending the entire source, just because the filename changed. > This would also take care of me moving files between directory trees, > and again do a mv instead of a delete/resend (if I rsync the root of > all that of course) > > Or is this possible already? This is not possible yet. It is on my wishlist for a future program. Of course remotely detecting files that have moved between directories might mean having the server hash every file on the filesystem. So it might be quite expensive... -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync // su
On 4 Sep 2003 Carson Gaspar <[EMAIL PROTECTED]> wrote: > --On Friday, September 05, 2003 12:45 PM +1000 Martin Pool > <[EMAIL PROTECTED]> wrote: > > > On 4 Sep 2003 Atom 'Smasher' <[EMAIL PROTECTED]> wrote: > > > >> obviously, allowing root logins through ssh (or any protocol, > >really)> is best avoided. > > > > Can you explain why you hold that opinion? > > Speaking as a security weenie, the problem is the utter lack of an > audit trail. "root" isn't a person, it's a role. If you allow direct > role logins, you have no idea what person is responsible. ... if you do the transitions to superuser in a way that doesn't record who did them, yes. Whether that transition occurs at login or later is incidental to whether we know who did it. The real problem seems to be that sudo/su emits a log message saying which user is becoming root, and ssh by default does not know who the remote user is. However, as you say, if you get SSH to log the public key which was used to become root,or use Kerberos, then you have quite good identification. Obviously password authentication is not enough; it could be anyone who knows the assword. The other root (heh) problem is that people are logging in as root when probably all they need is somethng like CAP_DAC_READ_SEARCH, the ability to read all files. Going to some kind of pam modules that set the right capabilities would be a better fix. This too requires no (or few) changes to rsync. > I don't, however, think that the rsync protocol is the right place to > fix it(speaking about normal rsync +rsh/ssh/whatever, not the rsync > daemon). Fixing the security issues with the daemon is a much more > difficult proposition. > > Possible options: > - Don't allow root to log in, require su, sudo, or a similar mechanism > (such as RBAC in Solaris). This makes rsync unhappy. It can easily be done by writing your own script which you call instead of ssh. Anything that ends up giving a socketlike connection will do. > - Allow cryptographically authenticated remote users (such as kerberos > roles - [EMAIL PROTECTED]) to log in as role accounts. Sadly, I don't > know of anything but kerberos that really does this well. SSH can do > something similar using RSA/DSA auth and logging the key fingerprint. Either of those sound good. Presumably you have sufficiently few people with root access that you can keep track of their keys. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync // su
Hi, Please send questions to the list, not to me. On 4 Sep 2003 Atom 'Smasher' <[EMAIL PROTECTED]> wrote: > i've been trying to find a good answer for this, but pretty much all > i've found is disagreement on what level of insecurity is > acceptable > > the problem arises when trying to use rsync as root. > > obviously, allowing root logins through ssh (or any protocol, really) > is best avoided. Can you explain why you hold that opinion? > of course, there are things that might need to be backed up > through rsync that can only be had with root permissions. > > there are several hacks i've found that play with known hosts and > their keys, or scripting which side initiates the transfer, etc. they > all try to bypass the problem, but most really don't fix it. > > here's my dream fix: > > add a new option to rsync:-su (The short-options -su already have meanings; you must mean --su.) > let's say i want to mirror the /etc directory, but leave root login > disabled... wouldn't it be cool if i could: > > rsync -var --delete -e ssh -su\ > [EMAIL PROTECTED]:/etc/ \ > /usr/backup/suspicious/etc/ > > and then, it would prompt me for the root passwd and (assuming that > i'm in the wheel group) continue the rsync as if i logged in as root. Can you please explain how sending a root password over an ssh channel is more secure than directly logging in as root? Indeed, for some cases (such as the user 'atom' being compromised) it seems less secure. If you really want to do this you can probably do it using an rsync wrapper script invoked through --rsync-path. > if i was any good in C, i'd at least prototype it for you. anyway, if > you think that would be a good addition to rsync, i know i'd enjoy > seeing it in a future release. > > anyway, maybe some of my open-source projects will be of help to > you http://business-php.com/opensource/ Regards, -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: GZIP, ZIP, ISO, RPM files and rsync, tar, cpio
On 28 Aug 2003 jw schultz <[EMAIL PROTECTED]> wrote: > On Thu, Aug 28, 2003 at 12:51:16PM +0300, Sviatoslav Sviridov/Lintec > Project wrote: > > > > Sorry for direct reply, but mail server at samba.org blocks my > > messages. Can you please tell me exactly what error you get, and preferably forward to me an example bounce? If sending it to me does not work either then please send it to jw and he can forward it to me. > > > Finally, Most distribution ISOs use package formats, such as > > > RPM, that compress the package contents. These compressed > > > packages may even if the installed fileset is unchanged > > > contain bits of meta-data that have been updated impacting > > > the rsyncabilty of the package file. In any case changing > > > even one internal file of a compressed package can disrupt > > > rsyncing the entire package file. The only possible > > > amelioration of this would be the use of the gzip > > > --rsyncable option (which requires a patched gzip) by the > > > package builders--assuming they use gzip for package > > > compression. Given the effect of improving rsyncability and > > > thereby reducing bandwidth requirements such a change to > > > their package build scripts could well be to their > > > advantage. > > > > BTW, is there patch for bzip2 that adds --rsyncable option? Or may > > bw someone working on it? > > I don't expect so. > > The --rsyncable patch for gzip uses file content patterns to > reset the compression algorithm so that even if you insert > or delete data early in the file rsync can still find > matching blocks. Look at the patch for further details. > > As far as i can tell from the manpage bzip2 is compresses > data in fixed size blocks with a reset on block boundaries. > This means that it is moderately rsyncable as long as you > never insert or delete data. You can change early data > without affecting later blocks but only if the offsets of > later blocks remain the same. This does not lend it to an > rsyncable patch. This does mean that bzip2 is good for > block oriented data such as database tablespace files and > for files that are appended to but bzip2 would be > undesireable for text, word processor, tar and other less > structured files. I think this is correct. You could naively simulate this by just splitting your file into 900kB chunks before compression. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: mknod / rsync error
On 22 Aug 2003 16:11:21 +0200 Lars Bungum <[EMAIL PROTECTED]> wrote: > Greetings! > > I'm experiencing these problems as described in this mail: > > --- > From: Thomas Quinot ([EMAIL PROTECTED]) > Subject: Rsync 2.5.5: FreeBSD mknod can't create FIFO's > This is the only article in this thread >View: Original >Format > Newsgroups: mailing.unix.rsync > Date: 2002-06-24 06:05:25 PST > The following patch (adapted to rsync 2.5.5 from the one posted in > Dec. 2000, > http://lists.samba.org/pipermail/rsync/2000-December/003349.html) n.b. the patch quoted in this mail was truncated. That looks reasonable to me. -- Martin GNU does not eliminate all the world's problems, only some of them. -- The GNU Manifesto pgp0.pgp Description: PGP signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync daemon and secrets file
On Mon, 25 Aug 2003 12:49:36 -0400 Hardy Merrill <[EMAIL PROTECTED]> wrote: > rsync -avv [EMAIL PROTECTED]::test-secret/one_secret > /tmp/rsync_test_secret Yes, that's better. > Although 'man rsync' does "technically" describe this > in the CONNECTING TO AN RSYNC SERVER OVER A REMOTE SHELL > PROGRAM section with this command: > > rsync -av --rsh="ssh -l ssh-user" [EMAIL PROTECTED]::module[/path] > local-path But that's not what you're doing here. You're just connecting to an rsync server over TCP. > IMHO, it would enhance user understanding to provide a > concrete EXAMPLE of this. Also, it would help in > 'man rsyncd.conf' not only to see an example of an > rsyncd.conf file, but also to see examples of the > different transfers that could be done with that > rsyncd.conf file. I'm not criticizing - just mearly > noticing an area that given some attention, could > increase user understanding and decrease support. Could you please draft a couple of paragraphs to add to the manual that you think would improve it? If you post them here I will check them and commit them. Thanks, -- Martin GNU does not eliminate all the world's problems, only some of them. -- The GNU Manifesto -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync silently changes special files to regular ones!
On Sun, 24 Aug 2003 16:44:15 +0200 clemens fischer <[EMAIL PROTECTED]> wrote: > rsync version 2.5.6 protocol version 26 on FreeBSD 4.8-STABLE i386 > "promotes" character special files to regular files: > > --8<---cut here:--start--->8-- > # ll /dev/stdout > 8282 crw-rw-rw- 1 root wheel - 22, 1 Aug 23 17:30 /dev/stdout > > # rsync localhost::rsync/readme /dev/stdout > $Id: readme,v 1.2 2003/08/05 02:38:25 root Exp root $ > ... > > # ll /dev/stdout > 7527 -rw-r--r-- 1 root wheel - 136 Aug 24 11:49 /dev/stdout > > # ./MAKEDEV std > > # ll /dev/stdout > 7527 crw-rw-rw- 1 root wheel - 22, 1 Aug 24 11:49 /dev/stdout > --8<---cut here:---end>8-- > > i know that this (displaying a text file on the console) is not what > rsync is meant to do, but since it didn't give any messages or exit- > codes, i didn't think of checking /dev/stdout. since then i chased a > phantom bug in freebsds update procedure for two weeks, never > suspecting an rsync bug. It is not "used on" special files. rsync replaces the destination with the source. It is just the same behaviour as mv. It is not like 'cat >dest'. [EMAIL PROTECTED]:/tmp# mknod pts5 c 136 5 [EMAIL PROTECTED]:/tmp# date>hello [EMAIL PROTECTED]:/tmp# ls -l hello pts5 -rw-r--r--1 root root 29 Aug 25 17:20 hello crw-r--r--1 root root 136, 5 Aug 25 17:20 pts5 [EMAIL PROTECTED]:/tmp# mv hello pts5 [EMAIL PROTECTED]:/tmp# ls -l hello pts5 ls: hello: No such file or directory -rw-r--r--1 root root 29 Aug 25 17:20 pts5 I can see how it would be confusing, but it is not a bug. > rsync should error exit or message the user when used on special > files! No, I don't think so. At the most I could stretch to giving a warning when replacing a special file, but even then I'm not sure. Unix is "you asked for it, you got it." Cheerio, -- Martin pgp0.pgp Description: PGP signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Current version for SCO 5.0.6
On Thu, 21 Aug 2003 16:01:50 -0400 "Patrick G. Davis" <[EMAIL PROTECTED]> wrote: > Does somebody have a current version of rsync already compiled for sco > openserver 5.0.6? Free software like rsync has no place in SCO's future. Ask SCO for help. -- Martin pgp0.pgp Description: PGP signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [librsync-devel] Re: state of the rsync nation? (revisited6/2003 from 11/2000)
On 8 Jun 2003, Donovan Baarda <[EMAIL PROTECTED]> wrote: > regarding librsync... It is still in sort-of-active development on > SourceForge by a variety of developers... a new release is waiting in > CVS for me to finally get around to releasing it, but I'm busy on a big > contract at the moment so its currently on hold pending some more > cygwin/win32 testing. It is in active use by projects like rdiff-backup. > AFAIK, rproxy is pretty much dead, and the only version that exists > depends on a very old version of libhsync. The closest thing to this > available now is the http proxy "proof of concept" with xdelta, but it's > radically different in many ways to the old rproxy (due to xdelta not > using signatures). The main reason why rproxy is dead is that dynamically-generated HTML files, where in principle rproxy does best, are just not a very important problem for many people at the moment. For users with ADSL or better HTML is not a problem, binaries may be. I realize there are people in the world who are still using modem connections where rproxy might be a win but I don't see any of them asking for commit access. A positive thing for rproxy is that both Mozilla and Apache2 are pretty stable now, and they have good interfaces for adding streaming compression support. Neither Apache1.2 or Netscape, which were dominant when I started on it could handle this very well. > This is largely still true, except libhsync changed back to librsync and > now has its own project on SourceForge separate from the mostly defunct > rproxy. librsync itself has no "wire format", being just a general > purpose signature/delta/patch library implementing the rsync algorithm. > > The comments about rsync never using libhsync/librsync are still true > for the foreseeable future. There are many things rsync includes that > are still missing from librsync, and the rsync implementation is very > tightly coupled, with many backwards compatibility issues. Even when > librsync reaches the point of being as good or better than rsync at > signature/delta/patch calculation, it would be a major task to "fit it > into" rsync. I think it's best at the moment to let rsync continue as a nice stable program, good at what it does. Wayne and myself have been toying with replacements in the background and perhaps in the future something better will come out, and perhaps it will use librsync. > rsync also has more active development, mostly in the form of > incremental feature additions and the resulting "bugfix fire-fighting", > all of which lead to an even more tangled implementation. Occasionally > there are efforts to re-write and clean up sections of the code, but > they are (rightly) regarded cautiously because of the breakage risk > involved for little immediate gain. > The librsync code in CVS is still largely "not very good". It is pretty > messy and needs a good cleanup. True. :-/ I think I got a bit mentally twisted up by trying to support nonblocking operation, which I still think is very important the the library being generally useful. Doing this in C is a bit hard. But it could certainly be done much better. The other thing I would really like to see is a more thorough test suite. > The API is mostly OK though, and it _does_ work quite well, with no > known bugs. I have some plans for a major cleanup and optimisation > of the code based on my experiences with pysync. I have a patch > submitted that I plan to commit after the next release that > optimises and cleans up the delta calculation code quite a bit. > > The "next big thing" in delta calculation is probably going to be the > vcdiff encoding format, which should allow a common delta format for > various applications and supports "self-referencing delta's", which > makes it capable of compression. According to the xdelta project this > has already been implemented, and I'm keen to see Josh's code, as it > could be used as the basis for a cleanup/replacement of at least the > "patch" component of librsync. Yes, that sounds good. > I think someone has a Perl wrapper for librsync that was being used > as a test bed for rsync 3 type development (superlifter?). "superlifter" was my prototype. It uses Python, and in fact just calls out to rdiff at the moment. At the moment I see it as another layer above librsync/rdiff that provides pipelined delta-compressed remote network IO, optionally over SSL or SSH. On top of this you could build a batch transfer like rsync 2.6, or an interactive client, or a backup system like Duplicity, or a real-time mirror based on dnotify. > For the future I can see continued support of the exising rsync code. I > would also like to see librsync adopt vcdiff as it's delta format, and > get a major cleanup, possibly by re-using some xdelta code. There are > many common elements to the xdelta and rsync algorithms, and I see no > reason why a single library couldn't support both (as pysync does). It > would be nice if librs
Re: patch draft for extended attributes on linux
On 25 Jun 2003, Hardy Merrill <[EMAIL PROTECTED]> wrote: > Martin Pool [EMAIL PROTECTED] wrote: > > > Nevertheless i do think it worth having something in > > > patches. > > > > Yes, if I can get this working I think the best place for it to end up > > is as an unofficial patch. > > Are these patches archived somewhere on the rsync website? They should be in the list archives. It would be nice to set up a little patch grabber. Please don't use that patch for anything other than decoration, it probably has bugs still. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: patch draft for extended attributes on linux
On 25 Jun 2003, Wayne Davison <[EMAIL PROTECTED]> wrote: > On Wed, Jun 25, 2003 at 10:34:38AM +1000, Martin Pool wrote: > > There is no mtime for xattrs, so they are transferred every time as > > part of the file list. > > One possibly better solution would be to create some kind of CRC of the > xattr data (MD4/MD5/whatever) and send just that in the file list for > each file. This would allow you to figure out when to update the xattr > data, but the protocol would need to be modified to send the xattr data > during the file-update phase (and possibly to allow the reciever to > request just an xattr update without doing a file update). That's a pretty good idea. For the moment I just wanted a minimal patch, as traffic size is not an overwhelming consideration for the particular user I was helping. However, for many realistic cases the xas are quite small. It is entirely possible for a file's attr and value them to be smaller than a 20-byte SHA1. (Well, perhaps not with my inefficient packing, but in principle they might be.) In cases where xattrs are used for security information, it might not be sufficient to apply them just at the end of the transfer. That might make the permissions on the temporary file too weak. Or perhaps not -- I just didn't want to think about it. :-) -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: patch draft for extended attributes on linux
On 24 Jun 2003, jw schultz <[EMAIL PROTECTED]> wrote: > That lack of an mtime for xattr could well cause > difficulties for backup systems as well. Perhaps a note to > the filesystems people is in order. The problem is that you > can't use mtime for these. It really needs its own > timestamp, perhaps as a mandatory system attribute. Yes, I think so. If that still makese sense when I'm finished on this I will send mail. Perhaps it would fit well into the reiserfs 'tree of small things' model of the world. > I don't much care for sending the xattrs as part of the file list. > Even the 4KB ext[23] _currently_ limit it to is huge. I'm not sure what is typical here. The situation I'm working on is replicating a Samba share which is storing ACLs and EAs in XFS EAs. Most of them will be pretty small, and most files won't have them. For a small tree with a short XA on each file and no other changes, it's like this: [data]$ ~/work/rsync/xa/rsync -aPzv distcc-2.7.1/ dest --xattr building file list ... 161 files to consider wrote 12365 bytes read 20 bytes 24770.00 bytes/sec total size is 1115540 speedup is 90.07 [data]$ ~/work/rsync/xa/rsync -aPzv distcc-2.7.1/ dest building file list ... 161 files to consider wrote 3027 bytes read 20 bytes 6094.00 bytes/sec total size is 1115540 speedup is 366.11 I think it's tolerable. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: patch draft for extended attributes on linux
On 24 Jun 2003, jw schultz <[EMAIL PROTECTED]> wrote: > I don't much care for sending the xattrs as part of the file list. > Even the 4KB ext[23] _currently_ limit it to is huge. I would have preferred to do it doing the regular transfer, rather than in the file list, but that seemed to make it a bit harder to ensure that the attributes were always applied even if the file was not otherwise modified, or if it were a symlink, etc. > Nevertheless i do think it worth having something in > patches. Yes, if I can get this working I think the best place for it to end up is as an unofficial patch. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
patch draft for extended attributes on linux
This draft patch adds support for transferring extended attributes
with a new --xattr option. It ought to work on Linux with XFS or
ext2/ext3 filesystems with the SGI/bestbits attribute system.
It is partially working, but there seems to be some kind of hang bug
while transferring the file list. I suspect it might be provoking a
problem in io.c.
You need to rerun autoconf, autoheader and configure after applying
it.
There is no mtime for xattrs, so they are transferred every time as
part of the file list. This means that they will be updated correctly
if you change attributes but do not change the file.
I wrote this because it was required by a colleague. I have mixed
feelings about whether this ought to be merged, even once it's working
correctly. rsync hardly needs more options or protocol
variations. :-( (Amusingly enough I once said "-xattr" instead of
"--xattr" and it silently did something else.)
diff -urpdN -x .ignore -x packaging -x cvs.log -x configure -x config.h.in -x
autom4te.cache -x config.log -x .cvsignore -x dummy -x .svn -x ID -x TAGS
rsync-2.5.6/Makefile.in xa/Makefile.in
--- rsync-2.5.6/Makefile.in 2003-01-21 05:26:14.0 +1100
+++ xa/Makefile.in 2003-06-24 15:08:09.0 +1000
@@ -34,7 +34,7 @@ OBJS1=rsync.o generator.o receiver.o cle
main.o checksum.o match.o syscall.o log.o backup.o
OBJS2=options.o flist.o io.o compat.o hlink.o token.o uidlist.o socket.o \
fileio.o batch.o clientname.o
-OBJS3=progress.o pipe.o
+OBJS3=progress.o pipe.o xattr.o
DAEMON_OBJ = params.o loadparm.o clientserver.o access.o connection.o authenticate.o
popt_OBJS=popt/findme.o popt/popt.o popt/poptconfig.o \
popt/popthelp.o popt/poptparse.o
diff -urpdN -x .ignore -x packaging -x cvs.log -x configure -x config.h.in -x
autom4te.cache -x config.log -x .cvsignore -x dummy -x .svn -x ID -x TAGS
rsync-2.5.6/cleanup.c xa/cleanup.c
--- rsync-2.5.6/cleanup.c 2003-01-27 14:35:08.0 +1100
+++ xa/cleanup.c2003-06-24 16:16:58.0 +1000
@@ -26,7 +26,7 @@
* shutdown() of socket connections. This eliminates the abortive
* TCP RST sent by a Winsock-based system when the close() occurs.
**/
-void close_all()
+void close_all(void)
{
#ifdef SHUTDOWN_ALL_SOCKETS
int max_fd;
diff -urpdN -x .ignore -x packaging -x cvs.log -x configure -x config.h.in -x
autom4te.cache -x config.log -x .cvsignore -x dummy -x .svn -x ID -x TAGS
rsync-2.5.6/configure.in xa/configure.in
--- rsync-2.5.6/configure.in2003-01-28 16:27:40.0 +1100
+++ xa/configure.in 2003-06-24 20:27:45.0 +1000
@@ -5,7 +5,7 @@ AC_CONFIG_SRCDIR([byteorder.h])
AC_CONFIG_HEADER(config.h)
AC_PREREQ(2.52)
-RSYNC_VERSION=2.5.6
+RSYNC_VERSION=2.5.6-xa
AC_SUBST(RSYNC_VERSION)
AC_MSG_NOTICE([Configuring rsync $RSYNC_VERSION])
@@ -267,6 +267,7 @@ AC_CHECK_HEADERS(glob.h mcheck.h sys/sys
AC_CHECK_HEADERS(netdb.h)
AC_CHECK_HEADERS(malloc.h)
AC_CHECK_HEADERS(float.h)
+AC_CHECK_HEADERS(attr/xattr.h)
AC_CHECK_SIZEOF(int)
AC_CHECK_SIZEOF(long)
@@ -414,6 +415,7 @@ AC_CHECK_FUNCS(waitpid wait4 getcwd strd
AC_CHECK_FUNCS(fchmod fstat strchr readlink link utime utimes strftime)
AC_CHECK_FUNCS(memmove lchown vsnprintf snprintf asprintf setsid glob strpbrk)
AC_CHECK_FUNCS(strlcat strlcpy strtol mtrace mallinfo setgroups)
+AC_CHECK_FUNCS(lgetxattr)
AC_CACHE_CHECK([for working socketpair],rsync_cv_HAVE_SOCKETPAIR,[
AC_TRY_RUN([
diff -urpdN -x .ignore -x packaging -x cvs.log -x configure -x config.h.in -x
autom4te.cache -x config.log -x .cvsignore -x dummy -x .svn -x ID -x TAGS
rsync-2.5.6/flist.c xa/flist.c
--- rsync-2.5.6/flist.c 2003-01-19 05:00:23.0 +1100
+++ xa/flist.c 2003-06-25 08:29:52.0 +1000
@@ -1,7 +1,7 @@
/*
Copyright (C) Andrew Tridgell 1996
Copyright (C) Paul Mackerras 1996
- Copyright (C) 2001, 2002 by Martin Pool <[EMAIL PROTECTED]>
+ Copyright (C) 2001-2003 by Martin Pool <[EMAIL PROTECTED]>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@@ -422,6 +422,12 @@ static void send_file_entry(struct file_
}
#endif
+#if SUPPORT_XATTRS
+if (opt_xattr) {
+xalist_send(f, file->xattrs);
+}
+#endif
+
#if SUPPORT_HARD_LINKS
if (preserve_hard_links && S_ISREG(file->mode)) {
if (remote_version < 26) {
@@ -457,7 +463,9 @@ static void send_file_entry(struct file_
}
-
+/**
+ * This matches up with send_file_entry()
+ **/
static void receive_file_entry(struct file_struct **fptr,
unsigned flags, int f)
{
@@ -555,6 +563,13 @@ static void receive_file_entry(struct fi
sanitize_path(file->link, file->dirname);
}
}
+
+#if SUPPORT_XATTRS
+if (opt_xattr) {
+xalist_receive(f, file);
+
Re: Smoother bandwidth limiting
On 15 May 2003, Paul Slootman <[EMAIL PROTECTED]> wrote: > I can't really see that doing smaller writes will lead to packets being > padded, unless you're doing really small writes (ref. the ATM 48-byte > packets); the TCP and IP headers will always be added, which means that > the extra overhead of those will have a larger impact than any > padding. > > So, I'd suggest that 1024 isn't that bad a number for all cases; it'll > fit comfortably into most MTU sizes, and for dialup PPP it'll be split > into two packets without that much overhead. If not concerned with the > dialup PPP case, I'd go for something like 1400. Of course a write() does not necessarily correspond to a TCP frame, which does not necessarily correspond to an IP packet. But nevertheless I would suggest avoiding writes that are this short. In addition to the headers that Paul mentioned, there are other per-packet costs such as Ethernet leadin and trailer times, and the hardware, interrupt and OS overhead for processing packets. Consider also that some people use rsync on fast networks, and they won't appreciate small packets *or* getting more system calls to process a given amount of data. Needlessly causing each packet hold 30% less data than it normally would is very wasteful. The point of bwlimit is after all to help users have more bandwidth for other applications. Checking for bwlimit after every say 4k I can imagine but below that is dubious. I'm happy to be proved wrong though. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Smoother bandwidth limiting
On 4 Feb 2003, jw schultz <[EMAIL PROTECTED]> wrote: > Yes but i'd like to hear from some people who know network > performance programming. I know only enough to be mildly dangerous. :-) I don't think you can do this optimally in userspace, because there is lots of buffering between what we write to the kernel and what gets onto the wire, which is generally what the user cares about. It will interact with the MTU, which is generally small enough not to matter, but also with the TCP window size. I think by throttling our connection we will also change the TCP window dynamic behaviour. In particular with no bwlimit rsync will often be blocked on network IO, but it may not be with bwlimit. This might make a difference to whether the Nagle algorithm comes in to effect to get packets pushed out. There is also some kind of interaction with routers with their own queues (as for ADSL, etc), and performance on fast networks may be very different. So I would be a bit cautious of applying patches based on one person's experience. Doing larger writes is likely to make the bandwidth more "jerky", as the kernel buffer is filled up, drains, and then pauses. That might make rsync's interaction with interactive traffic more harmful than it ought to be. But bringing it right down to 1024B doesn't sound good -- it's likely to generate http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Oops more testing was required....
On 18 Jun 2003, jw schultz <[EMAIL PROTECTED]> wrote: > On Wed, Jun 18, 2003 at 09:09:59PM +1000, Martin Pool wrote: > > On 17 Jun 2003, Rogier Wolff <[EMAIL PROTECTED]> wrote: > > > > > > Oops. Missed one line in the last patch > > > > Thankyou. That looks good. > > > > If we're going to make this more accurate it might be worthwhile to > > actually look at how long we really did sleep for, and use that to > > adjust time_to_sleep rather than resetting to zero. > > That would have to be a platform specific thing since not > all systems modify the timeout value to reflect the amount > of time not slept. Nevertheless that is a nice idea. Right, I know that is not portable but I forgot to say so. As Rogier say, you need to call gettimeofday() or some such. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Openssl and rsync
On 20 Feb 2003, Lee Wiltbank <[EMAIL PROTECTED]> wrote: > I have been working on a project to Openssl'ify Rsync. I am having > problems when Rsync forks two processes to handle a sender and was > wondering if anyone else would be able to lend a hand or some > pointers. I have posted to mailing.openssl.dev > > Basically, I have brought up Rsync 2.5.5 on cygwin. It goes through > the Openssl init and then waits on accept. When it accepts, it forks > itself. The child then communicates with the other side. The problem > occurs when Rsync determines that it is the receiver and forks itself > again. Then there are two processes with the same SSL object. I > am having a lot of trouble getting data from the client once Rsync > does this two-fork thing. I know that Openssl is multi-thread > capable, but this situation is where two processes have the same > SSL object and attached socket and are working with it. > > This version works fine when Rsync is the sender, that is, the client > connects and the server sends the files over. In this case, there is > only one process trying to write to the SSL object and socket. There are a few difficult historical facts, as you say - Forking on the receiver is pretty tightly embedded into the application and protocol. - OpenSSL just cannot handle a single socket being used from two different processes. Unlike with threads, it does not have any easy place to keep common state. And in any case the two processes assume they can just do their own thing; I'm not sure they would be able to synchronize in the way required by SSL. Consider that a "write" operation from the applications point of view may result in several reads and writes in the SSL layer, and vice versa. None of these issues are absolutely insurmountable (it's just code) but they are pretty tough. My recommendation is not to waste your time. Therefore one might look at putting SSL in a separate process that stands between the real socket and rsync. You could fork this off just before starting processing, perhaps in response to a command line parameter. Or you could just use stunnel, which does essentially this only with a bit more manual setup. SSH already works and is at least as easy to set up as SSL. (Key management is far more practical.) Is SSH not available on Netware, or are there issues with it? -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Oops more testing was required....
On 17 Jun 2003, Rogier Wolff <[EMAIL PROTECTED]> wrote:
>
> Oops. Missed one line in the last patch
Thankyou. That looks good.
If we're going to make this more accurate it might be worthwhile to
actually look at how long we really did sleep for, and use that to
adjust time_to_sleep rather than resetting to zero.
Also I'd prefer the variable be called micros_to_sleep or
us_to_sleep. Small point I know.
> diff -ur rsync-2.5.6.orig/io.c rsync-2.5.6/io.c
> +++ rsync-2.5.6/io.c Tue Jun 17 23:43:49 2003
> @@ -416,10 +416,19 @@
> * use a bit less bandwidth than specified, because it doesn't make up
> * for slow periods. But arguably this is a feature. In addition, we
> * ought to take the time used to write the data into account.
> + *
> + * During some phases of big transfers (file XXX is uptodate) this is
> + * called with a small bytes_written every time. As the kernel has to
> + * round small waits up to guarantee that we actually wait at least
> + * the requested number of microseconds, this can become grossly
> + * inaccurate. We therefore keep a cumulating number of microseconds
> + * to wait, and only actually perform the sleep when the rouding
> + * becomes insignificant. (less than 10%) -- REW.
> **/
> static void sleep_for_bwlimit(int bytes_written)
> {
> struct timeval tv;
> + static int time_to_sleep = 0;
>
> if (!bwlimit)
> return;
> @@ -427,9 +436,13 @@
> assert(bytes_written > 0);
> assert(bwlimit > 0);
>
> - tv.tv_usec = bytes_written * 1000 / bwlimit;
> - tv.tv_sec = tv.tv_usec / 100;
> - tv.tv_usec = tv.tv_usec % 100;
> + time_to_sleep += bytes_written * 1000 / bwlimit;
> +
> + if (time_to_sleep < 10) return;
> +
> + tv.tv_sec = time_to_sleep / 100;
> + tv.tv_usec = time_to_sleep % 100;
> + time_to_sleep = 0;
>
> select(0, NULL, NULL, NULL, &tv);
> }
--
Martin
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Multistreaming rsync
On 10 Feb 2003, "Cockram, Michael L (ISI)" <[EMAIL PROTECTED]> wrote: > Newbie here! > > I am not sure if this is possible or not, but is it possible to multistream > the connections that rsync is making? Say I had a directory with a bunch of > huge sized files. Is there a way of telling rsync to make multiple > connections for different groups of files? Am I making sence? Just run different rsync processes for different subdirectories. There is no support in the program itself. > Are there tcp window limitations on rync like ftp has? What do you mean? TCP windows are pretty much invisible to applications. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Feature request: true multiple sources
On 14 Jun 2003, Gregory Brauer <[EMAIL PROTECTED]> wrote:
>
> I am a big fan of rsync, but the more I use it, the more I
> become frustrated at rsync's asymetrical functionality.
> For instance, I can do this:
>
> rsync /A/ /B/ desthost:/AB
>
> but not this:
>
> rsync srchost:/A/ srchost:/B/ /AB
rsync allows remote shell wildcards:
rsync 'srchost:/{A,B}/' /AB
The limitations are in your own mind. (Well, at least this one
is. :-)
--
Martin
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: support@microsoft e-mails is a VIRUS
On 20 May 2003, jw schultz <[EMAIL PROTECTED]> wrote: > > > Is there anyway you can stop sending these e-mails to everybody on the list? > > > I've received maybe 3 or 4 of them since yesterday. > > > > One possible solution to reduce the spam/virus traffic on the list would > > be to close the list so that only people on the list can send to it. > > The rsync team has, so far, rejected that approach. We want > to keep the list as open as possible. Many people post to the list without subscribing, because it is the main support forum for a product. It is not really "closed" in the way that a list for a development team is. So there would be a lot of mails blocked. If they're automatically bounced then it is annoying for rsync users. If they're deferred then the delay is annoying, and somebody needs to spend time reading through the queue. At the moment I don't think that would be a good use of time. The only real solution is to send spammers and virus writers to jail. In the mean time we have set up spam and virus filters. As jw says, you are only seeing a small fraction of the literally hundreds of attacks we suffer every day. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Interactive Rsync Authentication Problem
On 29 May 2003, Andrew Klein <[EMAIL PROTECTED]> wrote:
> The getpassphrase() call is identical to getpass() except it returns 256
> chars maximum. Of course you would have to mess with autoconf but I
> don't think that should be too hard. Based on the autoconf stuff in the
> latest rsync release, the compile check would be something along these
> lines:
>
> AC_CACHE_CHECK([for getpassphrase],rsync_cv_HAVE_GETPASSPHRASE,[
> AC_TRY_COMPILE([#include ],
> [char *pass; pass = getpassphrase("Password: ");],
> rsync_cv_HAVE_GETPASSPHRASE=yes,rsync_cv_HAVE_GETPASSPHRASE=no)])
> if test x"$rsync_cv_HAVE_GETPASSPHRASE" = x"yes"; then
>AC_DEFINE(HAVE_GETPASSPHRASE)
> fi
Can you try that and tell us if it actually works?
It's OK if you can't get the autoconf stuff straight, but it would be
good to know that getpassphrase() actually solves the problem before
worrying.
Better yet, send a patch that adds an appropriately-licenced
readpassphrase()/getpassphrase() to the lib/ directory?
Someone wrote:
> I love the fact that the man page for getpass() under Linux says "don't use
> this", but does not provide any alternative. Mmmm... Linux - it's so
> secure! ;-)
Solaris fnmatch("ass", "hat", 0) used to return true!
--
Martin
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: You have emailed an address at dslreports.com
On 16 Jun 2003, Lapo Luchini <[EMAIL PROTECTED]> wrote: > Each time I send a message to the ML I receive this message... (thi > mislead me to double-post some days ago). > Could someone please unsubscribe the "blocked" address? > But I guess that's not possible, as anyone else shuold have noticed > this, too... =( Done. (I saw it too.) -- Martin pgp0.pgp Description: PGP signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [librsync-devel] Re: state of the rsync nation? (revisited6/2003 from 11/2000)
On 12 Jun 2003, jw schultz <[EMAIL PROTECTED]> wrote: > Mind you, that means making the server lightweight with the > client doing all the logic and a nearly stateless connection. > Much like my earlier post on this thread posited. I was wondering today if that would make it easier to gain confidence in the design's security. Making the semantics of an operation less dependent on a lot of accumulated state probably helps, all other things being equal. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [librsync-devel] Re: state of the rsync nation? (revisited6/2003 from 11/2000)
On 12 Jun 2003, jw schultz <[EMAIL PROTECTED]> wrote: > Leave the communications protocol to the communications > layer. You don't save anything by coding reordering and > retransmission at the packet level; that is infrastructure. > > Connectionless is fine. Lightweight sessions is better. If > you lose a connection a restart is possible. It is > preferable to not have to authenticate and negotiate > protocol versions and encryption with every message. > > Think in terms of transactions. Each transaction is atomic. > If a transaction doesn't complete you have the means to > roll-back and retry. If a connection breaks between > transactions, or leaving a transaction incomplete, you start > a new connection and pick up where you left off. I agree with all this. To extend on what jw says: I think it's fine to (if desired) negotiate SSL, authentication, and compression at the start of a connection. They generally require multiple round trips and it would be wasteful to do them more frequently when per-connection is natural. On the other hand it would be nice if the client could pick up an interrupted transfer halfway through the tree, rather than needing to start from the beginning as rsync 2.x does. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [librsync-devel] Re: state of the rsync nation? (revisited6/2003 from 11/2000)
On 13 Jun 2003, Martin Pool <[EMAIL PROTECTED]> wrote: > > Why run this _only_ over TCP? Obviously you don't want to re-invent TCP/IP > > error handling, but the protocol shouldn't rely on such a system. File > > transfer can potentially run connectionless. > > It sounds like you're talking about something like NFS (XDR-RPC) that > can run over UDP or TCP? Having said that Leaving out unnecessary dependencies is fine, but I wouldn't want to actually add new code, protocol or design features unless there was something it was useful for. I think if NFS was being done now, people would just use TCP and be done with it. The v4 spec recommends this. On modern hardware for this sort of application TCP performance is fine. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [librsync-devel] Re: state of the rsync nation? (revisited6/2003 from 11/2000)
On 12 Jun 2003, Brad Hards <[EMAIL PROTECTED]> wrote: > Hash: SHA1 > > On Wed, 11 Jun 2003 11:25 am, Martin Pool wrote: > > That could be a pretty nice thing. We use little rsync shares on > > workstations here for sharing files, and I know some people do the > > same with FTP. > > > > What aside from SLP would make this more useful? > A standardised way of describing the share would be good. By this, I don't > mean a software implementation, but a user / admin configuration. Think > Standard Operating Procedures. > The other thing that would be nice would be a search capability - "find me the > shares with a copy of rsync-2.5.6.tar.bz2" OK, interesting. > 1. I'm thinking about something that, as a minimum, doesn't do plain text > passwords. I admire clever attacks as much as the next guy, but the next guy > doesn't want some kewl hax0r with a copy of tcpdump uploading warez either. > Probably SASL is worth a look. Yes, SASL looks like the way to go, at least for authentication. Some things I read indicate that SASL is not a good choice for encryption/integrity. So perhaps we should use SASL just for authentication, and SSL for confidentiality/integrity. Does that make any sense? > Why run this _only_ over TCP? Obviously you don't want to re-invent TCP/IP > error handling, but the protocol shouldn't rely on such a system. File > transfer can potentially run connectionless. It sounds like you're talking about something like NFS (XDR-RPC) that can run over UDP or TCP? I wouldn't rely on TCP specifically, but I think it's OK to rely on a byte stream channel, such as TCP or SSH. I suppose if you're going to do UDP then you might want to try to do multicast too, but that makes things like error handling a lot harder. But I do think there should be a layer at which there are distinct messages, and that what goes under that might be something other than a byte stream in future. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [librsync-devel] Re: state of the rsync nation? (revisited6/2003 from 11/2000)
On 11 Jun 2003, Donovan Baarda <[EMAIL PROTECTED]> wrote: > On Wed, 2003-06-11 at 13:59, Martin Pool wrote: > > On 11 Jun 2003, Donovan Baarda <[EMAIL PROTECTED]> wrote: > > > > > The vcdiff standard is available as RFC3284, and Josh is listed as one > > > of the authors. > > > > Yes, I've just been reading that. > > > > I seem to remember that it was around as an Internet-Draft when I > > started, but it didn't seem clear that it would become standard so I > > didn't use it. > > I'm not sure if this is the same one... I vaguely recall something like > this too, but I think it was an attempt to add delta support to http and > had the significant flaw of not supporting rsync's > "delta-from-signature". It may have come out of the early xdelta http > proxy project. IMHO rproxy's http extensions for delta support were > better because they were more general. Yes, the most recent version of the Mogul delta-http proposal I read assumed that the server had a complete history of the document to generate diffs. This is fine if you're serving e.g. software distributions or content from a version control system and have the history, but not very general. > I forget if I saw this in Tridge's thesis, but I definitely noticed that > librsync uses a modified zlib to make feeding data to the compressor and > throwing away the compressed output more efficient. I have implemented > this in pysync too, though I don't use a modified zlib... I just throw > the compressed output away. Yes, I remember that, but that's not rzip. By the way the gzip hack is an example of a place where I think a bit of extra compression doesn't justify cluttering up the code. I think I'd rather just compress the whole stream with plain gzip and be done. See http://samba.org/~tridge/phd_thesis.pdf pg 86ff rzip is about using block search algorithms to find widely-separated identical blocks in a file. (I won't go into detail because tridge's explanation is quite clear.) I am pretty sure you could encode rzip into VCDIFF. I am not sure if VCDIFF will permit an encoding as efficient as you might get from a format natively designed for rzip, but perhaps it will be good enough that using a standard format is a win anyhow. Perhaps building a VCDIFF and then using bzip/gzip/lzop across the top would be acceptable. In fact rzip has more in common with xdelta than rsync, since it works entirely locally and can find blocks of any length. rzip's advantage compared to gzip/bzip2 is that it can use compression windows of unlimited size, as compared to a maximum of 900kB for bzip2. Holding an entire multi-100MB file in memory and compressing it in a single window is feasible on commodity hardware. > The self referencing compression idea is neat but would be a... > challenge to implement. For it to be effective, the self-referenced > matches would need to be non-block aligned like xdelta, which tends to > suggest using xdelta to do the self-reference matches on top of rsync > for the block aligned remote matches. Fortunately xdelta and rsync have > heaps on common, so implementing both in one library would be easy (see > pysync for an example). > > If I didn't have paid work I would be prototyping it in pysync right > now. If anyone wanted to fund something like this I could make myself > available :-) I may get a chance to work full time on replication again soon, so I'm trying to work out where we're up to. > Yeah, my big complaint about librsync at the moment is it is messy. Just > cleaning up the code alone will be a big improvement. I would guess that > at least 30% of the code could be trimmed away, leaving a cleaner and > more extensible core, and because "messy" leads to "inefficient", it > would be faster too. "If I'd had more time this letter would have been shorter." -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [librsync-devel] Re: state of the rsync nation? (revisited6/2003 from 11/2000)
On 11 Jun 2003, Donovan Baarda <[EMAIL PROTECTED]> wrote: > The vcdiff standard is available as RFC3284, and Josh is listed as one > of the authors. Yes, I've just been reading that. I seem to remember that it was around as an Internet-Draft when I started, but it didn't seem clear that it would become standard so I didn't use it. > I also had some correspondence with Josh ages ago where he talked about > how self-referencing delta's can directly do compression of the miss > data without using things like zlib and by default gives you the > benefits of rsync's "context compression" without the overheads (rsync > runs a decompressor _and_ a compressor on the receiving end just to > regenerate the compressed "hit" context data). Something possibly similar is mentioned in tridge's thesis. I was talking to him a while ago and (iirc) he thought it would be good to try it again, since it does well with the large amounts of memory and CPU time that are available on modern machines. I strongly agree with what you said a while ago about code simplicity being more valuable than squeezing out every last bit. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [librsync-devel] Re: state of the rsync nation? (revisited6/2003 from 11/2000)
On 10 Jun 2003, Brad Hards <[EMAIL PROTECTED]> wrote: > Yep. Also, I was playing with the idea of rsync with Service Location Protocol > to use as a replacement for the crappy practice of sharing data over floppy > disks. The rough concept was that each machine had a shared directory, which > you could conveiently label and advertise over SLP. That could be a pretty nice thing. We use little rsync shares on workstations here for sharing files, and I know some people do the same with FTP. What aside from SLP would make this more useful? > Go superlifter! For what it is worth, the things I identified during the > abortive kioslave / SLPv2 share development: > 1. More secure than FTP. > 2. Easy to label shares/directories and provide fine grained access control, > if desired. > 3. Client side library that doesn't require hellish text parsing, or at least > hides it from you. > 4. Well delimited packets, so you can tell when one has been > dropped. Can you give more detail on those? What do you mean by packets being dropped? How can that happen on a TCP channel? -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: state of the rsync nation? (revisited 6/2003 from 11/2000)
On 8 Jun 2003, Donovan Baarda <[EMAIL PROTECTED]> wrote: > The "next big thing" in delta calculation is probably going to be the > vcdiff encoding format, which should allow a common delta format for > various applications and supports "self-referencing delta's", which > makes it capable of compression. According to the xdelta project this > has already been implemented, and I'm keen to see Josh's code, as it > could be used as the basis for a cleanup/replacement of at least the > "patch" component of librsync. Do you have a link for this? Josh plays his cards pretty close to his chest. The XDelta page seems to be even more inactive than librsync :-/ -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: state of the rsync nation? (revisited 6/2003 from 11/2000)
On 9 Jun 2003, Brad Hards <[EMAIL PROTECTED]> wrote: > Hash: SHA1 > > On Sun, 8 Jun 2003 15:43 pm, Donovan Baarda wrote: > > The comments about rsync never using libhsync/librsync are still true > > for the foreseeable future. There are many things rsync includes that > > are still missing from librsync, and the rsync implementation is very > > tightly coupled, with many backwards compatibility issues. Even when > > librsync reaches the point of being as good or better than rsync at > > signature/delta/patch calculation, it would be a major task to "fit it > > into" rsync. > The downside to not having a library that is wire-compatible with rsync > - --daemon is that it is damn difficult to write something that works as a VFS > / kioslave type device. I had a hack at this, by wrapping the rsync > executable, and it worked a bit, but it was way too fragile for any real use: > http://www.cuneata.net/rsync-kio.html I guess the reason why you're interested in doing it is so that you can browse public rsync mirrors from Konqueror/whatever? Speaking only for myself, I don't think this is worth spending time on. It would be hard to write a wire-compatible library, and hard to refactor rsync into such a library. Not only might a new tool be written more easily without baggage, it might also (in a couple of years) persuade people running mirror sites to switch. I know many of them are unhappy with rsync at the moment: - large memory usage - no really good ways to restrict client usage - ... -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: ACL Support? Where can I get it?
On 18 Apr 2003, Justin Kreger <[EMAIL PROTECTED]> wrote: > From what I've read on the list, there does not appear to be POSIX ACL > support for rsync at this time, is this true? There does however appear > to be a patch floating around out there, where can I find this > patch. http://www.lpmd.org/rsync/ -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
(fwd) PATCH: managing permissions with rsyncd.conf options
-- Martin --- Begin Message --- This is a patch to control unix permissions when uploading to a rsyncd-server by setting rsyncd.conf options. cu, Stefan -- Stefan Nehlsen | ParlaNet Administration | [EMAIL PROTECTED] | +49 431 988-1260 rsyncd.conf options to handle file permissions (stolen from samba) This patch is made to provide more control on the permissions of files and directories that are uploaded to a rsyncd-server. Normally when files and directories are uploaded to a rsyncd they are created with the permissions of the source. Especially in the case that user and group are set to special values using the uid and gid directives it does not much sense to use the source permission pattern. There is a patch introducing a new chmod command line option but normally you may want to control the permissions on server side. The patch below will allow you to modify file and directory permissions by using 4 new rsyncd.conf directives. I'm sure that those 2 patches will not break each other and it really makes sense to use them both. You may know this options from samba :-) create mask When a file is created (or touched) by rsyncd the permissions will be taken from the source file bit-wise 'AND'ed with this parameter. This parameter may be thought of as a bit-wise MASK for the UNIX modes of a file. Any bit not set here will be removed from the modes set on a file when it is created. The default value of this parameter is set to 0 to be provide the default behaviour of older versions. Following this rsync will bit-wise 'OR' the UNIX mode created from this parameter with the value of the force create mode parameter which is set to 000 by default. This parameter does not affect directory modes. See the parameter "directory mask" for details. See also the "force create mode" parameter for forcing particular mode bits to be set on created files. See also the "directory mask" parameter for masking mode bits on created directories. Default: create mask = 0 Example: create mask = 0644 force create mode This parameter specifies a set of UNIX mode bit permissions that will always be set on a file created by rsyncd. This is done by bitwise 'OR'ing these bits onto the mode bits of a file that is being created or having its permissions changed. The default for this parameter is (in octal) 000. The modes in this parameter are bitwise 'OR'ed onto the file mode after the mask set in the "create mask" parameter is applied. See also the parameter "create mask" for details on masking mode bits on files. Default: force create mode = 000 Example: force create mode = 0644 directory mask When a directory is created (or touched) by rsyncd the permissions will be taken from the source directory bit-wise 'AND'ed with this parameter. This parameter may be thought of as a bit-wise MASK for the UNIX modes of a file. Any bit not set here will be removed from the modes set on a file when it is created. The default value of this parameter is set to 0 to be provide the default behaviour of older versions. Following this rsync will bit-wise 'OR' the UNIX mode created from this parameter with the value of the "force directory mode" parameter which is set to 000 by default. This parameter does not affect file modes. See the parameter "create mask" for details. See also the "force directory mode" parameter for forcing particular mode bits to be set on created directories. See also the "create mask" parameter for masking mode bits on created files. Default: directory mask = 0 Example: directory mask = 0755 force directory mode This parameter specifies a set of UNIX mode bit permissions that will always be set on a directory created by rsyncd. This is done by bitwise 'OR'ing these bits onto the mode bits of a directory that is being created. The default for this parameter is (in octal) which will not add any extra permission bits to a created directory. This operation is done after the mode mask in the parameter "directory mask" is applied. See also the parameter directory mask for details on masking mode bits on created directories. Default: force directory mode = 000 Example: force directory mode = 0755 diff -ur rsync-2.5.5/loadparm.c rsync-2.5.5-umask/loadparm.c --- rsync-2.5.5/loadparm.c Mon Mar 25 05:04:23 2002 +++ rsync-2.5.5-umask/loadparm.cSun Mar 2 22:53:16 2003 @@ -140,6 +140,10 @@ int timeout; int max_co
(fwd from kladit@t-online.de) files of length zero
- Forwarded message from Klaus Dittrich <[EMAIL PROTECTED]> - From: [EMAIL PROTECTED] (Klaus Dittrich) Subject: files of length zero Date: Tue, 11 Mar 2003 17:08:47 +0100 To: [EMAIL PROTECTED] User-Agent: Mutt/1.4i X-Bogosity: No, tests=bogofilter, spamicity=0.00, version=0.10.2 Hi Martin, MS-Windows users here sometimes make the expierience to become files of length zero when something on windows crashes. They often have many files open and after a crash they don't realize that parts of their work gots lost. Nightly a backup-server using rsync, copies those zero length files and thereby destroyes their files backed up the day before by makeing it zero length too. Can you build in an option to rsync that handles files of length zero the same way as deleted ones, so preserving the old file ? -- Regards Klaus - End forwarded message - -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync in-place (was Re: rsync 1tb+ each day)
On 4 Feb 2003, jw schultz <[EMAIL PROTECTED]> wrote: > The reason why in-place updating is difficult is that > rsync expects the unchanged blocks in the old file may be > relocated. Data inserted into or removed from the file does > not require the rest of the file to be retransmitted. > Unchanged blocks will be copied from the old locations in > the old file to new locations in the new file. > > In-place updates requires that blocks not relocate. > It may be possible by disallowing matches having differing > offsets. That would require deeper investigation. Of course the other place where people want this is for transfers of block devices, where the rename is just not possible. I looked a little at doing this in librsync. The naive solution is to merely prohibit the delta from referring to blocks that have been already overwritten. I will probably eventually add at least this option. You might try this in rsync. A lot of other code to do with e.g. setting permissions makes the assumption of the rename model, though. It would take a fair amount of testing. Of course this model really falls down in some cases. Consider the case of one block inserted at the beginning. Then with the naive "no backreferences" approach every block will be overwritten just before it's needed. :( You can imagine a smarter algorithm that does non-sequential writes to the output so as to avoid writing over blocks that will be needed later. Alternatively, if you assume some amount of temporary storage, then it might be possible to still produce output as a stream. Really for your problem the practical solution is just to dump the whole file, perhaps allowing for sparse blocks. As other people have observed, by design rsync does a lot more disk IO than network. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: reconnect ssh connection?
On 30 Jan 2003, David Garamond <[EMAIL PROTECTED]> wrote: > has someone come up with a trick to let disconnected ssh connections be > recovered without terminating and having to restart rsync (perhaps by > wrapping ssh or something)? Ooh, interesting idea... You might do it with some kind of wrapper at both ends... Alternatively, by changing ssh options perhaps you can get the process to stay open even if the link goes away, by increasing timeouts and so on... -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: proposal to fork the list (users/developers)
On 30 Jan 2003, "Green, Paul" <[EMAIL PROTECTED]> wrote: > I tend to be someone who automatically looks for trends, and the nice thing > about having just one list is that it lets me know where people are having > problems. Judging by the number of questions we get, one of the biggest > challenges for inexperienced rsync users is knowing why a particular file is > included or excluded. Yes, that's definitely a large advantage of having a single list. > Way in the back of my mind I see a need for an option that, for > every file included or excluded, says which rule was used to make > the decision. Nice and simple. I came to the same conclusion in a similar way a while ago. If you use -vv for rsync, you should see messages about exactly this. :-) -- Martin Debian: giving you the power to shoot yourself in each toe individually. -- ajt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [trivial patch] link overloaded
On 29 Jan 2003, jw schultz <[EMAIL PROTECTED]> wrote: > This is just a trivial documentation change. The word > "link" is overloaded. It refers to symlinks, hardlinks and > network links. When looking for references to file links in > the manpages the network references get in the way. +1 -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Proposal that we now create two branches - 2_5 and head
On 30 Jan 2003, Donovan Baarda <[EMAIL PROTECTED]> wrote: > On Thu, 2003-01-30 at 07:40, Green, Paul wrote: > > jw schultz [mailto:[EMAIL PROTECTED]] wrote: > > > > [general discussion of forthcoming patches removed] > > > > > All well and good. But the question before this thread is > > > are the changes big and disruptive enough to make a second > > > branch for the event of a security or other critical bug. > > > > Agreed. > [...] > > After reading arguments, is support the "delay the branch till it > absolutely must happen" approach... ie don't branch until a bugfix needs > to go in to a "stable" version and HEAD is way too "unstable" to be > released with the fix as the new "stable". Yes, that is generally a better approach. Remember, you can always go back and create a branch from the release later on if such a situation occurs. Personally, I'm more interested in eventually starting from scratch with something like duplicity, rzync, or superlifter. I think the way Subversion builds on the experience but not the code from CVS is pretty good. Obviously there are downsides to this approach: it may be a long time before the code is ready, and people may not want to switch for a while after that. But it may be more fun, and eventually yield a cleaner solution. I hope other people are interested in continuing work on librsync and projects based on it. I think the parallels between rsync and CVS are actually reasonably strong: - good tools, and de facto standards both for the free software community - showing signs of age in underlying assumptions (file-by-file versioning in CVS, shared filelist in rsync) - knotty code and interface that are a bit hard to refactor - most existing users have it working properly and don't *want* disruptive changes, just bug fixes or perhaps small additional features - new approach offers substantial benefits - doing something new is not urgent All the above is just for me personally. Continuing to move rsync itself forward as and when appropriate is still a good thing. > Actually, a bigger "attitude" issue for me is having a separate > rsync-devel and rsync-user lists. I have almost unsubscribed many times > because of the numerous newbie user questions; Me too. Samba does this with samba-technical and samba. I think at this point the user list for samba only has slightly more traffic than rsync. I think apache may now be the same too. Plenty of people post user questions to samba-technical despite prominent notices that it is only for developers. They tend to both piss off developers and go unanswered at least some of the time. It's probably due both to "my question *is* technical" and "if the developers read it they might answer." I'm not sure what a good solution would be: probably a clearer name would help. Perhaps rsync-dev? What do people feel about this? > I'm only interested in the devel stuff. I'm sure there are many > users who have unsubscribed because of the numerous long technical > posts. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Proposal that we now create two branches - 2_5 and head
On 28 Jan 2003, "Green, Paul" <[EMAIL PROTECTED]> wrote: > I think splitting the branches will also let us be a little more > experimental in the development branch, at least until we get near > the next release phase, because we'll always have the field release > in which to make crucial bug fixes available quickly. I agree that this would be a good approach if and only if there is energy to do lots of development in the head branch. What do you have in mind? -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
list filtering
Because of the enormous amount of traffic being generated by Windows viruses[0] I have turned on Mailman attachment filtering on the high-traffic samba.org lists. Lists will now pass only text/plain MIME parts through to the list. multipart/alternative messages with both text and html forms will have the HTML form removed, and messages in only HTML will be squashed to text. Messages which cannot be handled in any of these ways will be rejected. To send patches or log files to the list, you need to either insert them inline into your message, or make sure they're marked as text/plain. On most systems, just making the name be *.txt should be sufficient. I hope everybody's enjoying their SQL Server experience :-) -- Martin samba.org postmaster [0] ... automated notifications about viruses, users complaining about viruses, users complaining about automated notification, users complaining about users complaining, scanners complaining about perfectly ordinary attachments, etc -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: signing tarballs
[replied to list] There was a discussion about this on the Samba list a while ago http://lists.samba.org/pipermail/samba-technical/2002-November/040931.html Briefly We should create a team signing key, with an lifetime of about a year. It has to be relatively short to allow for turnover in the people who have access to the key. The signing key must only be stored on secure machines, certainly *not* on samba.org.(If it was on samba.org, somebody who compromised that machine could also generate new signatures and it would be pointless.) The key should be signed by team members and other relevant people; we should also sign each others' keys. The key should be on the keyservers and on the web site. Unless you've already done so I'll create the key and send the private half to you and the public half to the website, keyservers, and list. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: SPAM on List...
On 10 Dec 2002, jw schultz <[EMAIL PROTECTED]> wrote: > First let me say that Martin (and any others list managers) > is doing pretty well. Although there was a breif rise in > the volumen of spam leaking through during the transition > it has settled down quite nicely. This is an arms war and > I don't expect perfection. Cudos! Thanks! > I can almost second that. That seems to hold true for the > last couple of months. Perhaps html is already blocked. > I do know that some valid mail may come in with > Content-Type: Multipart/Alternative where one is text/plain. > Although i don't like the waste of bandwidth i could see > accepting that. It is the stuff that is only html that > should definitely be bounced. I've wondered about installing something like mimedefang to handle these things. It would be nice to get rid of TNEF attachments too. I won't start this until we have some experience with the new stopspam-bogofilter setup. There are some complications: As Tim points out, some people don't control whether their mailer sends HTML or not. So we would need to fall back to html->text conversion, rather than bouncing such messages. This makes it not a good way to detect spam. Some people need to send patches/log files/whatever to the lists as attachments. "What's not there can't break." Unless it's clearly useful, it shouldn't be installed. Given that some people can't change their HTML setup (not under their control or too clueless) I'm not sure if notification messages are useful. > The other clear indicator that comes up more often here > seems to be non-english messages. Care has to be taken not > to block just because of a few words but if the message is > mostly non-english or is in a charset incompatible with > english it should be bounced. The previous bouncer did explicitly block non-latin character sets. However, there was a nasty failure mode which caused some non-junk messages to be blocked. People writing from (say) China may be using a mail client that sends messages in a Chinese character set. Some of those character sets contain latin characters, so they may have in fact been writing a purely English message, or perhaps an English message with a part-Chinese sig block. Discarding these messages was incorrect; what was worse was that the old system gave no indication of how to fix the problem and the messages were dropped without review. :-( As an amusing example of going too far in the other direction, a certain government body has "XXX" as a blackword in their mail filter, and a single occurrence is enough to cause the messages to bounce. Of course people pretty regularly write "XXX" for "don't care" values... And let's not even think about byte sex. :-) -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: SPAM on List...
On 9 Dec 2002, "John E. Malmberg" <[EMAIL PROTECTED]> wrote: > I will agree that the SAMBA lists are being kept more spam free than > some of the other mail servers that I get e-mail on. Just as an interesting data point: our bogofilter setup caught 60 spam messages in the last 24 hours aimed at lists.samba.org lists. > And while you are saying that you are not in favor of using blocking > lists, you are blocking Korea by some method, but that could be just > something that bogofilter has figured out. We're using korea.services.net. Unfortunately the spam:ham ratio for Korea is so bad that this seems to be the only appropriate solution. We check the headers on samples of rejected messages and there are dozens of spams per day and I haven't seen a nonspam message yet. > It is your servers and your decisions on how to allocate your resources. > > No spam blocking method is 100%. > > And I am not complaining about your efforts. I was just posting some > methods of spam blocking in use, and of course my bias opinions on > them. Thanks, understood. If I'm defensive it's only because maintaining these things is generally a thankless task. People (not you, John) complain and whine when spam gets through, but nobody sees the work that goes into keeping the other 99% out and keeping things running smoothly. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: SPAM on List...
On 9 Dec 2002, "John E. Malmberg" <[EMAIL PROTECTED]> wrote: > If it was on any of the reputable blocking lists, I would not be able to > receive any of the SAMBA lists, and you would be getting the > bounces. It has since been removed from some of them. > I.P. based blocking has shown to be the only thing that motivates some > domains to act on abuse reports. I really don't care about abuse reports anymore. There is an inexhaustible supply of other spam sources. Desirable as it may be to have ISPs behave properly, it will not reduce the amount of spam. > And the bounce message can contain an alternate contact means such > as a web form if someone needs a white-listing. A major goal of this exercise is to reduce or eliminate the number of messages that require manual handling because they waste admin time, and they are often dropped. Our previous experience was that IP blacklists have significant false-positive and false-negative rates. In addition, IP blacklists seem to often "go mad" when the admins start pursuing a campagin against some ISP in a way that does not agree with our goals. For example, the previously-reputable ORBS server blacklisted most of Australia a few years ago. Basically I want the decisions to be made by samba team admins, not by other people. > Some time last fall apparently Korea passed an OPT-OUT with the > equivalent of "ADV" in the headers law. Right after that, list that I > subscribe to at a major university went from 2 spams a week to over 8 > spams a day. 99% from Korea. We no longer accept any mail from Korea. :-( > Now the other thing to consider is that when the filter makes a mistake > and deletes a legitimate message, it is quite a while before the sender > figures out, if at all that the message did not get through. Our filter sends intelligible, actionable bounce messages. This is an enormous improvement of the previous system, which said something like "error 10". > If the message is bounced, the sender knows immediately, and can use the > alternate contact information, such as a web form to request a > whitelisting. As RFC 2822 requires, mail to postmaster is not filtered, and is read by a human. People can report problems there. > They also know that there is probably a problem with their ISP or > with the particular block list, and they have the information needed > to fix it. That's bogus. If my ISP is blocked it is very difficult for me to change -- at home I am on a 12 month contract with my DSL provider, for example. Even if I did move, it's very unlikely that my leaving would persuade them to change/enforce their AUP. People with business hosting are in a even more difficult situation. > Filtering makes spam your problem. Using a blocking list makes spam the > problem of the ISP sending the spam. Eventually almost noone will > accept e-mail from them, either from local blocking lists, or public > ones. You describe a long-term solution in which spam-friendly ISPs are gradually ostracised. I'm not quite sure I believe you that there is a clear distinction, that bonafide ISPs are really able to stop spam, and that being ostracised will ever really cut them off. But regardless, these are long-term, global measures.What I care about is reducing admin load and spam transmission on samba.org right now. Our bogofilter setup seems to be doing *extremely well* at just that; I can see it catching many more messages and getting far fewer false positives before, and it is no longer necessary to clear queues by hand. I looked through the queue when I installed it and there were many posters who just happened e.g. to be from China and whose messages were basically dropped. Unless people have specific complaints about the new setup I intend to keep going along this path. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync] Re: bug reporting.. bugzilla
On 9 Dec 2002, R P Herrold <[EMAIL PROTECTED]> wrote: > Really a better FAQ editor process seems more useful. Isn't > this the purpose of a CVS and commit privileges -- set up one > or more trusted editors with rights, and delegate that aspect. Anybody who wants to maintain the FAQ-O-Matic already has the necessary access. If somebody starts working on it and feels that CVS would be more appropriate then of course we can switch to that. > For the last year, I have acted as editor on the RPM website; > there is also an open editorial mailing list, and provided > content (dreadfully little) gets slotted in. > > I monitor all the mailing lists in the area (five primary > ones) and watch for common questions or misunderstandings > which are well answered end up summarized and on the site. I > particularly look for the postings by the lead maintainer and > a few others for the 'nuggets' -- the answers float by and may > be picked out of the stream and tossed up on the riverbank of > a FAQ Yes, that's the process I had in mind. It's just a matter of some set of people finding the time and motiviation to do it. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: bug reporting.. bugzilla
On 5 Dec 2002, Sriram Ramkrishna <[EMAIL PROTECTED]> wrote: > What we do in the GNOME project is to find volunteers to run triage > and catalog the bugs. If you have a "bugmaster" position who could > coordinate something like this. The situation is rather different to GNOME, as jw notes: the code is not growing very quickly, and there are far fewer developers available to work on it. > > There used to be another bug reporting system but it was being ignored > > so Martin turned it off. Yes, I did. I think the system was broken by some kind of infrastructure migration, and since nobody seemed to use it I put up this page rather than fixing it: http://rsync.samba.org/nobugs.html One problem was that it used tridge's Jitterbug system, which is a nice program but a bit harder to learn than Bugzilla, or at least less familiar to most people. Also, because it runs mostly over email, it quickly fills up with spam. But the main thing that discouraged me from maintaining it was just that most of the entries were not valid bugs. We had large numbers of - misunderstandings of how to use rsync (operator error) - massively incomplete reports (e.g. just "it fails", without any error message.) - architectural limitations (e.g. upfront scan) - other junk entries and in addition many of them were redundantly reported. I think probably >90% of entries were like this. You can see this to a lesser extent in the FAQ-O-Matic: http://rsync.samba.org/fom-serve/cache/223.html http://rsync.samba.org/fom-serve/cache/39.html http://rsync.samba.org/fom-serve/cache/233.html Too many people fail to realize that filing a useful bug is actuallly a lot of work and requires that the reporter actually put a bit of thought into the problem. So the database was full of things that were not really bugs, which made it pretty useless either for people who wanted to find out about a bug they might be experiencing, or developers wanting to know how many bugs there are. I'm sure GNOME has had this too, but if I understand correctly they reduced their junk bug count in the first place by throwing out the whole database, and then by putting a lot of work into triage and cleaning. I think a better way go forward would be for volunteers to help maintain an FAQ. This might be a good way to address common problems, whether they result from misunderstandings or from program errors. It could be in FAQ-O-Matic or something else. Since new bug reports are relatively rare, but problems and misunderstandings seem to occur repeatedly I think this would be the most useful way to get all the information in one place. So who's interested in working on that? -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: SPAM on List...
On 15 Nov 2002, Tim Potter <[EMAIL PROTECTED]> wrote: > On Thu, Nov 14, 2002 at 09:05:27PM -0500, John E. Malmberg wrote: > > > The SAMBA-TECHNICAL list reported that they have gone to the > > bl.spamcop.net blocking list, and it has been relatively spam free since > > then. The bl.spamcop.net is an aggressive blocking list with a quick > > trigger. > > We did start using spamcop for a while but there was way to much > collateral damage inflicted on innocent parties. For example we missed > several offers of free hosting for the samba.org main server. And dp.samba.org (aka lists.samba.org) has an IP that is blacklisted by some people... IP-based blacklisting is too coarse a tool, and it makes it hard to make exceptions for people who really are not spammers, even if initially classified as such. News reports say that spam has more than tripled in the last year, which seems anecdotally true. I think we're actually getting more accurate classification, it's just that the numbers are larger. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: SPAM on List...
On 14 Nov 2002, Rainer Zocholl <[EMAIL PROTECTED]> wrote: > So the "list owner" should see thru (after spam filtering) the > remaining messages "on hold". > > That would be very nice. Mailman holds some suspicious messages for filtering by the admin. However, for samba.org, this means about 80 messages per week, which have to be handled through a clunky web interface, and which take time away from more useful tasks. This is not acceptable. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Head & Rotor VE 12/08A
On 8 Dec 2002, [EMAIL PROTECTED] wrote: > Howdy... > > > Can we get RID of this member? This is the 2nd time I have seen this > posted. Now after the first time, I figured it would have been put into a > SPAM filter, and thereby the member would not be able to post SPAM to the > list again, but that does not seen to be the case. We're working on improving spam filtering for the list using bogofilter. At the moment we catch about 100 spams per day going to the samba lists, so the percentage is not so bad. The only real solution is to jail spammers. > I still suggest we go to a closed list, whereby e-mail addresses are > verified by a person before being allowed to post. It would help with SPAM, > and when a member posts SPAM, they are put into moderated mode, and if they > do it a 2nd time, they are banned...permanently. samba.org already has lots of trouble with people who are not able to follow simple instructions about how to subscribe, unsubscribe, post, etc. Going to a closed list would cause more administrative work, and would also inconvenience posters who want to e.g. read via a local list. So at the moment I don't want to do that. -- Martin Open a medium-sized can of Spam (retain the can (retain the spam too)) -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
(fwd from david@interactiveinstitute.se) Bugs in rsync
- Forwarded message from David Jonsson <[EMAIL PROTECTED]> - From: David Jonsson <[EMAIL PROTECTED]> Subject: Bugs in rsync Date: Fri, 19 Jul 2002 18:38:59 +0200 (CEST) To: Martin Pool <[EMAIL PROTECTED]>, Andrew Tridgell <[EMAIL PROTECTED]> First, Thansk for a great tool! I run rsync supplied with RedHat 7.3 rsync --version rsync version 2.5.4 protocol version 26 Copyright (C) 1996-2002 by Andrew Tridgell and others <http://rsync.samba.org/> Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, IPv6, 64-bit system inums, 64-bit internal inums and I experience the followinf errors when I issue this command rsync -a --delete * /mnt/navhdb2 1. Files beginning with . will not get deleted at the destination if they don't exist in the source. (I detected that i leaves erased .forward files) 2. I hade replaced a directory with a symbolic link at the source but the destination kept the directory. Please respond to me just so I know the reasons of my problems. David - End forwarded message - -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync and locking
On 8 Nov 2002, Zas <[EMAIL PROTECTED]> wrote: > Hello, > > I got an issue with rsync and (fcntl) locked files. > > When rsync tries to sync a file that is opened and locked by another > process (that use fcntl write lock), it ends up in Disk Sleep. > Then i need to reboot to remove that dead processes (let me now if > there's another solution). Can you please see if you can attach strace to find out what it's tryig to do? Advisory locks should not cause a program to hang in 'D'. Are you sure you're not using mandatory locks? -- Martin "Those who are familiar with Open Source Software and Linux are favorably predisposed towards them." -- Microsoft white paper, http://www.opensource.org/halloween/halloween7.php -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: spam filter on rsync list
On 7 Nov 2002, jw schultz <[EMAIL PROTECTED]> wrote: > I trust we are in the training phase :) It was pre-seeded with some data, but there were some glitches in the integration with Mailman. Hopefully it's better now. -- Martin "Those who are familiar with Open Source Software and Linux are favorably predisposed towards them." -- Microsoft white paper, http://www.opensource.org/halloween/halloween7.php -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
spam filter on rsync list
# 0.40 0.40 156.153.255.237 # 0.40 0.307692 delivered-to # 0.40 0.228571 for # 0.40 0.164948 from # 0.40 0.116364 lists.samba.org # 0.40 0.080706 mbp # 0.40 0.055292 nov # 0.40 0.037553 palrel12.hp.com # 0.40 0.025353 postfix # 0.40 0.017046 received # 0.40 0.011429 return-path # 0.40 0.007648 rsync # 0.40 0.005112 samba.org # 0.40 0.003414 wed # 0.40 0.002278 with In response to rampant abuse, I have installed a new spam filter, Bogofilter, on the rsync mailing list. Experiments have indicated that it should get a smaller rate of false negatives or positives than the existing system. If there are any problems, please mail me or the postmaster. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: 2.5.6 release
On 5 Nov 2002, jw schultz <[EMAIL PROTECTED]> wrote: > This might be a good time for tagging 2.5.6 perhaps. A fair > number of bugfixes have gone in, popt updates, and a few new > features. It has been stable for about 2 months. Unless > there is something in the pipeline it sounds like time to > release and start on 2.5.7cvs. Sounds good to me. I'll do a 2.5.6pre to check next week, unless somebody else really wants to do it. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Disk Space Problems with Rsync.
On 23 Sep 2002, Bull Barry - bbull <[EMAIL PROTECTED]> wrote: > I have a problem with rsync causing a filesystem to think that it is 100% > used when actually it isn't. A quick reboot will cause the filesystem to go > to about 70%-75% used. Below is the script that I am executing. I am using > Red Hat Linux release 7.1 (Seawolf) Kernel 2.4.9-34smp on a 2-processor > i686. Does anyone have any idea on what causes this and how to fix > this? There is probably a file (or severral) that has been unlinked by rsync, but is still open by some other program, and therefore it's space can't be freed up. Killing the other relevant program, or rebooting, frees up the space. Use lsof to discover what files are open in that directory. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
rsync FAQ-O-Matic fixed
The rsync FAQ-O-Matic was broken during the migration of samba.org to new hardware. It has now been fixed. Please let me know if you see any other problems. http://rsync.samba.org/fom-serve/cache/1.html -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync-like ftping (addendum)
On 4 Aug 2002, Zoltan HERPAI <[EMAIL PROTECTED]> wrote: > sorry, just a small addendum , why this is all about :) i can't run an > rsyncd, since the site is hosted on a serverfarm, i have only ftp, http > and mysql ;) Ask your provider to install rsync. You are the customer, after all. Failing that, use one of the zillions of perl scripts people have already written to do ftp mirroring. http://freshmeat.net/search/?q=ftp+mirror§ion=projects -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: superlifter design notes and a new proposal
I think there was some confusion earlier in the thread about the "redo" thing in rsync 2. It's not for handling files that have changed during the transfer. My understanding of this is that it is used when the whole-file md4 hash shows that the block checksum actually made a mistake in transferring the file. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: superlifter design notes and a new proposal
On 4 Aug 2002, Wayne Davison <[EMAIL PROTECTED]> wrote:
> Your previous proposal sounded quite a bit more fine-grained than what
> rZync is doing. For instance, it sounded like you would have much more
> primitive building-block messages and move much of the controlling
> smarts into something like a python-language scripting layer. While
> rZync allows ftp-level control (such as "send this file", "send this
> directory tree", "delete this file", "create this directory") it does
> this with a small number of higher-level command messages.
OK, good.
> I think that's a good idea. My rZync app currently operates on each arg
> independently, but I recently discovered that this makes it incompatible
> with rsync when merging directories and such. For instance, the command
> "rsync -r dir1/ dir2/ dir3" merges the file list and removes duplicates
> before starting the transfer to dir3.
This is a substantial source of cruft in the current code, and one of
the reasons claimed to make an up-front traversal necessary.
I think a more efficient, and possibly simpler solution, would be to
first examine all of the source directories and determine their
relationships. Basically, you might discover that dir2 is in fact a
subdirectory of dir1, or the same (or vice versa), in which case you
can eliminate it. Or you might discover that they're disjoint. Given
that directories are trees, I don't think any there are any other
possibilities.
Doing this in a way that properly respects various symlink options
will be a little complex, but I think it is in principle possible. It
is also something quite amenable to being thoroughly exercised in
isolation as a unit test.
I am pretty sure that you can do this by just examining dir1 and dir2.
You do need to look at the filesystem to find out about symlinks and
so on, but I think you do not need to traverse their contents.
It is pretty complex, so there might be some case I've missed.
> I got rid of the "multi-IO" idiom of rsync in favor of sending all
> data via messages and limiting each chunk to 32K to allow other
> messages to be mixed into the middle of a large file's data-stream
> (such as verbose output).
OK, that makes sense. I guess 32k is as good a number as any.
> I think the basic idea of how rZync envisions a new protocol working is
> a good one -- not so much the specifics of the bytes sent in the
> message-header format, but how the messages flow, how each side handles
> the messages in a single process, how all I/O is handled by a single
> function, etc. There's certainly lots of room for improvement,
> though.
I've started looking at the code, and it looks very nice. It's
certainly easier to read that rsync. Would you mind putting in some
more comments to help me along though?
I had a couple of "internal" thoughts about how the code for a next
release ought to go. Please don't take them as criticisms of your
right to write experimental code however you want, or as an attempt to
dictate how we run things. I just want to raise the issues.
Global names should be distinguished with some kind of prefix, as in
librsync: "rz_" or whatever. If this ever turns into a library that
gets linked into something else it will help; in the meantime it helps
keep clear what is part of the project and what's pulled in from
elsewhere.
I really liked mkproto.awk when I first saw it, but now I'm not so
keen. I think maintaining header files by hand is in some ways a
good thing, because it forces you to think about whether a particular
function really needs to be exported to rest of the program, or to the
world at large.
>From rzync.h:
> #define MSG_HELLO 1
> #define MSG_QUIT 3
> #define MSG_NO_QUIT_YET 4 // XXX needed??
> #define MSG_ABORT 5
> #define MSG_NOTE_DIRNAME 6
> #define MSG_NOTE_FILENAME 7
> #define MSG_DEC_REFCNT8
These might work better as an enum, so that gdb can show symbolic
values.
> typedef struct {
> char *names[MAX_ID_LIST_LEN];
> long nums[MAX_ID_LIST_LEN];
> int count;
> } ID;
Linus has a rule about not using typedefs for structures, because it's
good to be clear about whether something is a structure or whatever.
I'm inclined to agree. So I would refer to that thing "struct rz_id"
or something.
Being 64-bit clean probably implies declaring rz_time_t, rz_uid_t and
so on, and using that rather than the native types, which will be
pretty random.
> This also reminds me that I hadn't responded to jw's question about why
> I thought his pipelined approach was more conducive to a batch protocol
> than an interactive protocol. To make the pipelined protocol as
> efficient as rsync will require the complexity of his backchannel
> implementation, which I think will be harder to get right than a
> single-process message-oriented protocol. If every stage is a separate
> process, it seems less clear how t
Re: superlifter design notes and a new proposal
I've been thinking a bit more about Wayne and jw's ideas. My first draft was proposing what you might call a "fine-grained" rpc system, with operations like "list this directory", "delete this file", "calculate the checksum of this file." I think Wayne's rzync system was kind of like that too. One unusual feature of rsync compared to most protocols is that a single request causes an enormous amount of stuff to happen: there is only one request/response per connection at the moment, really. It is a very CISC-like protocol. I wonder what we could achieve if we stay broadly within that model, of both parties knowing about the whole job, and working in tandem, rather than one of them controlling the other per file? So the client will send something more or less equivalent to its whole command line. This would be a more conservative design in some ways, because it is more similar to the existing system. It also perhaps avoids some of the issues about pipelining that have been giving me trouble at least. While staying with that overall approach, we may still be able to make some improvements in - documenting the protocol - doing one directory at a time - possibly, doing librsync deltas of directories - just one process on either end - getting rid of interleaved streams on top of TCP - sending errors as distinct packets, including a reference to the file that caused them (if any) - handling ACLs, EAs, and other "incidental" things - holding the connection open and doing more operations afterwards What made me start thinking this way is the realization that the basic idea of cooperating processes (rather than client-server) is not really causing us any trouble at the moment. Other things in that list are, like the interleaved error stream, or the 3-process model. But perhaps sending the arguments across the network and having the remote process know what to do is not such a problem. I will try to write up a more detailed description of this idea later on. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: new rsync release needed soon?
On 31 Jul 2002, Dave Dykstra <[EMAIL PROTECTED]> wrote: > Yes I think a new release is needed soon, but there's more patches than > that that should get in. We need to weigh up getting functions in vs making steps small enough that the chance of breakage is acceptable. I am afraid that at the moment our only means of getting really good cross-platform test coverage for rsync is to throw a release out, and so that inclines me towards being conservative in what we put in. Hopefully we can try to get people on the list testing -rc releases more aggressively. > A bunch of them have been posted and I was hoping you were keeping > track of them and would be putting more of them in. I will try to read back through the list and see about merging them this week, with a view to a release candidate on about the 11th, and a release about a week after that. > The patch that I'd most like to see get in JD Paul's patch for using SSH > and daemon mode together. We still don't have an agreement on what the > syntax should be. I think the combination of -e ssh and :: which he > implemented is the most understandable syntax and we should just go with > it. I agree that it would be really good to support it. However, -e and :: seem to be a persistent source of confusion for new users. I'm not sure if this change will help those people, or what if anything would be better. (More later on this.) -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: new rsync release needed soon?
On 1 Aug 2002, Dave Dykstra <[EMAIL PROTECTED]> wrote: > Another change that I think really ought to go in is something like > the one at > > http://lists.samba.org/pipermail/rsync/2002-February/006371.html > > to get the correct error codes out of rsync. But first I think we > really need to hear from Tridge why he put that code there in the first > place. Martin, did you ever ask him? If not, can you please get him > to look at it? I will follow that up with him. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Useless option combos (was Re: --password-file switch)
On 30 Jul 2002, Wayne Davison <[EMAIL PROTECTED]> wrote: > On Tue, 30 Jul 2002, Martin Pool wrote: > > The --password-file option only applies to rsync daemon connections, > > not ssh. > > Perhaps we should make rsync complain about such options that don't make > sense (another example being trying to use -e with a "::" hostspec)? There's a patch in cvs to make it complain about -e with "::". The manual actually already says that --password-file does not effect remote shells, but I have made it a bit more obvious. I agree that a warning would be good. Shall we do a new release soon? There's just one more change I would like to put in, which is partially rolling back the IPv6 patch so that it uses the old code, unmodified, if --disable-ipv6 is specified. I'm not sure this needs to go in before the next release though. I think it would reduce the overall level of pain, particularly on older platforms. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: --password-file switch
On 30 Jul 2002, Jochen K?chelin <[EMAIL PROTECTED]> wrote: > How can I use the --password-file switch with rsync in order not to > be promted for the users password so I can run rsync in a cronjob? > > rsync -uavrpog -e ssh /www [EMAIL PROTECTED]:/DESTINATION/`date +%A` > --password-file=/quellen/RSYNC_PASSWD > > does not work! > > I always get a prompt to enter users root password! The --password-file option only applies to rsync daemon connections, not ssh. You need to set up an ssh key to make ssh connections with no password; see the recent thread or the ssh manual for instructions. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: timestamp on symlink
On 29 Jul 2002, Donovan Baarda <[EMAIL PROTECTED]> wrote: > This is because most of python's os.xxx methods de-reference symlinks. You > get this error because 'nothere' doesn't exist. The correct way to get time > info on symlinks is to use os.lstat(), which doesn't de-reference links. I realize you can get the time that way (although not on all platforms), but how do you set it? As jw says, there is no lutime(). -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: timestamp on symlink
On 28 Jul 2002, Michael Wang <[EMAIL PROTECTED]> wrote:
> rsync does not sync the timestamp on symlink (Solaris 8).
>
> It is probablly due to the limitation of Unix implementation
> of symlink, but I would like to know why rsync/Unix does not
> do this, and what we can do about it. Is the conclusion that
> "rsync syncs everything except the timestamp on symlink"?
I don't think it is possible to set the time of a symbolic link. A
quick experiment on Linux seems to confirm that:
!891 12:36 ~% python2.2
Python 2.2.1 (#1, May 3 2002, 23:19:03)
[GCC 2.95.4 20011002 (Debian prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.symlink('nothere', '/tmp/mylink')
>>> os.utime('/tmp/mylink', None)
Traceback (most recent call last):
File "", line 1, in ?
OSError: [Errno 2] No such file or directory: '/tmp/mylink'
The Sun manpage doesn't comment on how they think this ought to work,
so presumably they're trying to update the link's target too.
You will need to do a backup below the filesystem API (e.g. using
dump) if you want to preserve them, as far as I can see.
> Why do I need timestamp on symlink? Supposed something stopped
> working because something removed the link, and then remake
> the link. If all I have is the rsync copy, then I would have
> a wrong conclusion, for example, that "nobody touched the directory
> since last night."
You will see that the directory's modification time has changed; you
will also see that the new symlink has a later time than that in the
backup.
> [root@emily:/tmp]ls -ls a/copying b/copying
>8 lrwxrwxrwx 1 root other 7 Jul 28 12:09 a/copying -> COPYING
>8 lrwxrwxrwx 1 root other 7 Jul 28 12:07 b/copying -> COPYING
>
> [root@emily:/tmp]rsync -avn /tmp/a/ /tmp/b
> building file list ... done
> wrote 25424 bytes read 20 bytes 16962.67 bytes/sec
> total size is 15972287 speedup is 627.74
>
> [root@emily:/tmp]rsync -av /tmp/a/ /tmp/b
> building file list ... done
> ./
> wrote 25424 bytes read 20 bytes 16962.67 bytes/sec
> total size is 15972287 speedup is 627.74
>
> [root@emily:/tmp]ls -ls a/copying b/copying
>8 lrwxrwxrwx 1 root other 7 Jul 28 12:09 a/copying -> COPYING
>8 lrwxrwxrwx 1 root other 7 Jul 28 12:07 b/copying -> COPYING
What is this example meant to demonstrate?
--
Martin
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: superlifter design notes (OpenVMS perspective)
On 27 Jul 2002, jw schultz <[EMAIL PROTECTED]> wrote: > The server has no need to deal with cleint limitations. I > am saying that the protocol would make the bare minimum of > limitatons (null termination, no nulls in names). It probably also makes sense to follow NFS4 in representing paths as a vector of components, rather than as a single string with '/'s in it or whatever. ['home', 'mbp', 'work', 'rsync'] avoids any worries about / vs \ vs :, and just lets the client do whatever makes sense. I don't know a lot about i18n support, but it does seem that programs will need to know what encoding to use for the filesystem on platforms that are not natively Unicode. On Unix it probably makes sense to default to UTF-8, but latin-1 or others are equally likely. This is independent of the choice of message locale. I think the W32 APIs are defined in Unicode so we don't need to worry. Quoting, translating, or rejecting illegal characters could all make sense depending on context. I guess I see John's backup vs distribution question as hopefully being different profiles or wrappers around a single codebase, rather than different programs. Perhaps the distinction he's getting at is whether the "audience" for the client who uploaded the data is the same client, or somebody else? -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: superlifter design notes (was Re: Latest rZync release: 0.06)
I'm inclined to agree with jw that truthfully representing time and leap seconds is a problem for the operating system, not for us. We just need to be able to accurately represent whatever it tells us, without thinking very much about the meaning. Somebody previously pointed out that timestamp precision is not a property of the kernel, but rather of the filesystem on which the files are stored. In general there may be no easy way to determine it ahead of time: you can (if you squint) imagine a network filesystem with nanosecond resolution that's served by something with rather less. I suspect the only way to know may be to set the time and then read it back. You can also imagine that in the next few years some platform may change to a format that accurately represents leap seconds, whether by TAI or something else. (I'm not sure if I'd put money on it.) Presumably that machine's POSIX interface will do a lossy conversion back to regular Unix time to support old apps. If we merely used that information, then when replicating between two such machines, files whose mtime happened to near on a leap second would be inaccurate. That would contradict our goal of preserving precision as much as possible, even if we can't tell if it is accurate. Ideally, we would use the native interface so as to be able to get the machine's full precision, and that would imply something like TAI internally. Whether this is worth doing depends on whether you reckon any platform will actually move to a filesystem that can represent leap seconds. As jw says, practically all machines have clocks with more than one second of inaccuracy, so handling leap seconds is not practically important. Certainly they might use it within their ntp code, but I don't know if they'll expose it to applications. > What is the actual format of TAI? 64-bit signed seconds-since-1970, plus optionally nanoseconds, plus optionally attoseconds. (There's something rather fascinating about using attoseconds.) To be fair, it seems that TAI is an international standard, and djb just made up libtai, not the whole thing. (Mind you, from some standards I've seen, that would be a good reason to walk briskly away.) One drawback, which is not realy djb's fault, is that if you inadvertently use a TAI value as a Unix value it will be about 10 seconds off -- almost, but not quite, correct. I'd hate to have bugs like that but presumably they can be avoided by using the interface correctly. On the other hand, sint32 unix time is clearly running out, and if we have to use something perhaps it might as well be TAI. I would kind of prefer just a single 64-bit quantity measured in (say) nanoseconds, and compromise on being able to time the end of the universe, but I don't think I care enough to invent a new standard. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: superlifter design notes (was Re: ...
On 27 Jul 2002, "John E. Malmberg" <[EMAIL PROTECTED]> wrote: > A program serving source files for distribution does not need to be that > concerned with preserving exact file attributes, but may need to track > suggested file attributes for for the various client platforms. > > A program that is replicating for backup purposes must not have any > loss of data, including any operating specific file attributes. > > That is why I posted previously that they should be designed as two > separate but related programs. I'm not sure that the application space for rsync really divides neatly into two parts like that. Can you expand a bit more on how you think they would be used? -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
