Re: include-exclude patterns

2002-12-10 Thread Paul Faure
This is why regular expression matching is 1000 times more powerful.
I submitted a patch a few months ago.

marco ([EMAIL PROTECTED]) wrote:
 Le Mon, 9 Dec 2002 21:22:48 -0800
 jw schultz [EMAIL PROTECTED] ?crivait :
 
 I found this on the FAQ-O-Matic from the rsync webite
 
 http://rsync.samba.org/fom-serve/cache/164.html
 
 I'm so agree...
 Is it possible to make it true ?
 
 thanks 
 
 -- 
 marco
 --
 Cl? PGP publique : https://iftbqp.mine.nu/marco.asc



-- 
Paul N. Faure BEng  613.266.3286
Carleton University Systems Eng paul-at-faure-dot-ca
Chief Technical Officer, CertainKey Inc.paul-at-certainkey-dot-com
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Re: Speed problem

2002-11-11 Thread Paul Faure
On Mon, 11 Nov 2002, jw schultz wrote:

 On Mon, Nov 11, 2002 at 04:30:05PM +0100, [EMAIL PROTECTED] wrote:
  Mermgfurt !
 
  I have some problem with syncing two machines which are connected
  over a Gigabit-connection. I'm trying to use rsync with ssh because of
  the authorisation mechanisms (keys). It starts quite ok with 18 MB/s
  (this small speed may have something to do with our internal net)
  and falls down to 400 KB/s (!!!). This happens over a long period
  because those files I want to copy are very big (upto 70 GB per file).
  Even though I tried to increase the blocking size the speed just goes
  down and won't go up again. In fact it really writes 18 MB/s, it's not
  just a problem of -partial or something similar. Ok, I haven't tried
  it without ssh yet, but it really looks very strange.
  The version is the rsync 2.5.6cvs version from debian-unstable.
 
  Thanks for any help !


Try it without ssh.

ssh may be waiting in the random pool for more entropy (randomness).
When it grabs a lot of random data, it must wait for more random things
to happen to populate the random pool, if it did not do this, your random
data would be predictable and thus insecure.

Try `ls -R /*` on your system when it slows down.

-- 
Paul N. Faure BEng  613.266.3286
Carleton University Systems Eng paul-at-faure-dot-ca
Chief Technical Officer, CertainKey Inc.paul-at-certainkey-dot-com

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



Regular Expression support

2002-11-11 Thread Paul Faure
I have added regular expression support using a POSIX implementation.

The patch (against 2.5.5) is attached.

The implementation is simple and follows the same mechanism that is
implemented for normal searches.

I added these command line arguments:
 --rexclude=PATTERN  exclude files matching regexp PATTERN
 --rexclude-from=FILEexclude regexp patterns listed in FILE
 --rinclude=PATTERN  don´t exclude files matching regexp PATTERN
 --rinclude-from=FILEdon´t exclude regexp patterns listed in FILE

In order to follow the previous format, the following will also work:
 --include +R PATTERN
 --include -R PATTERN
 etc...

I use this method to send the search patterns to a remote rsync server. So
if the pattern is a regular expression, three bytes are prepended to the
pattern (instead of two), a '+' or a '-', a capital 'R' and a space. A
check is done against the remove server version number to check if it
supports regular expressions.

The regular expression PATTERN supports two formats, the first is plain
and simple:
 .*\.txt$

The second format was to support some search options:
 /.*\.txt$/i

This allows for case insensitive (//i) matching and extended (//e)
matching.

The implementation is optimized so that it does not compile the regular
expression patterns more than once, for this reasons, a regex_t* pointer
was added to exclude_struct.

The man pages has been updated with the new format and a few examples.


-- 
Paul N. Faure BEng  613.266.3286
Carleton University Systems Eng paul-at-faure-dot-ca
Chief Technical Officer, CertainKey Inc.paul-at-certainkey-dot-com

Only in rsync-2.5.5.mod/: Makefile
Only in rsync-2.5.5.mod/: access.o
Only in rsync-2.5.5.mod/: authenticate.o
Only in rsync-2.5.5.mod/: backup.o
Only in rsync-2.5.5.mod/: batch.o
Only in rsync-2.5.5.mod/: checksum.o
Only in rsync-2.5.5.mod/: cleanup.o
Only in rsync-2.5.5.mod/: clientname.o
Only in rsync-2.5.5.mod/: clientserver.o
Only in rsync-2.5.5.mod/: compat.o
Only in rsync-2.5.5.mod/: config.h
Only in rsync-2.5.5.mod/: config.log
Only in rsync-2.5.5.mod/: config.status
Only in rsync-2.5.5.mod/: connection.o
Common subdirectories: rsync-2.5.5/doc and rsync-2.5.5.mod/doc
diff -u rsync-2.5.5/exclude.c rsync-2.5.5.mod/exclude.c
--- rsync-2.5.5/exclude.c   Mon Feb 18 14:10:28 2002
+++ rsync-2.5.5.mod/exclude.c   Mon Nov 11 14:50:00 2002
 -31,7 +31,7 
 static struct exclude_struct **exclude_list;
 
 /* build an exclude structure given a exclude pattern */
-static struct exclude_struct *make_exclude(const char *pattern, int include)
+static struct exclude_struct *make_exclude(const char *pattern, int include, int 
+regexp)
 {
struct exclude_struct *ret;
 
 -45,16 +45,25 
} else if (strncmp(pattern,+ ,2) == 0) {
ret-include = 1;
pattern += 2;
+   } else if (strncmp(pattern,-R ,3) == 0) {
+   pattern += 3;
+ret-use_real_regexp = 1;
+   } else if (strncmp(pattern,+R ,3) == 0) {
+   ret-include = 1;
+ret-use_real_regexp = 1;
+   pattern += 3;
} else {
ret-include = include;
+ret-use_real_regexp = regexp;
}
 
ret-pattern = strdup(pattern);
 
if (!ret-pattern) out_of_memory(make_exclude);
 
+if (!ret-use_real_regexp) {
if (strpbrk(pattern, *[?)) {
-   ret-regular_exp = 1;
+   ret-fnmatch = 1;
ret-fnmatch_flags = FNM_PATHNAME;
if (strstr(pattern, **)) {
static int tested;
 -76,6 +85,7 
if (!strchr(ret-pattern,'/')) {
ret-local = 1;
}
+}
 
return ret;
 }
 -85,15 +95,86 
free(ex-pattern);
memset(ex,0,sizeof(*ex));
free(ex);
+if(ex-regexp){
+  regfree(ex-regexp);
+  free(ex-regexp);
+}
 }
 
+#include regex.h
 static int check_one_exclude(char *name, struct exclude_struct *ex,
  STRUCT_STAT *st)
 {
char *p;
int match_start=0;
char *pattern = ex-pattern;
+// return 1; -- MEANS EXCLUDE
+printf(NAME=%s, %d %d\n, name, ex-use_real_regexp, S_ISDIR(st-st_mode));
+if (ex-use_real_regexp) {
+  #define reg_errbuf_size 256
+  char errbuf[reg_errbuf_size];
+  int status;
+  char *fullName=NULL;
+  if(!ex-regexp){ // Compile the regular expression only once, this speeds 
+up performance considerably
+int options = 0;
+char *modifiedPattern=NULL,
+ *optionsString=NULL;
+rprintf(FINFO, Compiling regular expression '%s'\n, pattern);
+ex-regexp=malloc(sizeof(regex_t));
+if(!ex-regexp) out_of_memory(check_one_exclude);
+memset(ex-regexp, 0, sizeof(regex_t));
+