Crazy excludes
I'm stuck!!! Look this command line log: -bash-2.05b# pwd /usr/backups -bash-2.05b# ls .backup.gaia.sh.swp backup.gaia.sh -- #NEXT COMMAND IS ENTERED BY HAND IN THE SHELL --- -bash-2.05b# rsync -a -v --delete --timeout 120 --exclude='/msurdi/*' localhost:/home/ /usr/backup s/gaia.0/ Password: receiving file list ... done created directory /usr/backups/gaia.0 ./ msurdi/ test/ test/aaa.txt sent 75 bytes received 195 bytes 77.14 bytes/sec total size is 0 speedup is 0.00 -bash-2.05b# ls gaia.0/msurdi/ --- #THAT'S RIGHT, NOTHING ON /HOME/MSURDI WAS TRANSFERRED :-) - --- #NOW, LET'S TRY FROM MY SCRIPT - -bash-2.05b# sh backup.gaia.sh Try 1 of 3... #NEXT COMMAND, IS JUST AN ECHO OF THE COMMAND THAT WILL BE RUN IN NEXT STEP. - rsync -a -v --delete --timeout 120 --exclude='/msurdi/*' localhost:/home/ /usr/backups/gaia.0/ Rotating backups... -bash-2.05b# rm -r gaia.0 --- #NOW, I REMOVED THE ECHO FROM THE SCRIPT SO THAT IT GETS RUN - -bash-2.05b# sh backup.gaia.sh Try 1 of 3... Password: receiving file list ... done created directory /usr/backups/gaia.0 ./ msurdi/ msurdi/.cshrc msurdi/.login msurdi/.login_conf msurdi/.mail_aliases msurdi/.mailrc msurdi/.profile msurdi/.rhosts msurdi/.shrc test/ test/aaa.txt sent 253 bytes received 4649 bytes 1400.57 bytes/sec total size is 3925 speedup is 0.80 Rotating backups... -bash-2.05b# --- #DAMN!DAMN!DAMN! - IF THE COMMAND IS EXACTLY THE SAME, ON THE SAME HOST ¿WHY RSYNC BEHAVES DIFFERENTLY IF IT'S RUN FROM THE COMMAND LINE OR FROM A SHELL SCRIPT? I'M THINKING THAT MAYBE COULD BE SOMETHING RELATED TO ANY ENVIROMENT VARIABLE. IF YOU WISH, I COULD POST THE ENTIRE SHELL SCRIPT SOURCE -- Thanks everybody in advance. Matias. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync with bwlimit
Hi, Any one give me some sample code with bwlimit option. what's is the std/ recommended value for this? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync help needed...
Matt McCutchen wrote: On Mon, 2006-02-27 at 06:58 -0800, lsk wrote: Could you give an example with syntax for rsync using file option --files-rom=FILE. If my-list in the current directory contains a b b/c b/d b/d/e , then the command rsync -a src/ dest/ --files-from=my-list copies src/a - dest/a src/b - dest/b src/b/c - dest/b/c src/b/d - dest/b/d src/b/d/e - dest/b/d/e . Rsync copies just those files and directories, preserving their attributes. It leaves alone any other files that might be in src/b, src/b/d, dest/b, or dest/b/d; even if --delete is given, it does not delete anything. Search the man page for --files-from for more information: http://rsync.samba.org/ftp/rsync/nightly/rsync.html Matt, Typically with Oracle databases, you will have datafiles residing on different mount points starting from the root such as: /p02/oradata/OSID/redo01.log /p03/oradata/OSID/redo02.log /p04/oradata/OSID/redo03.log /p01/oradata/OSID/system01.dbf /p04/oradata/OSID/undotbs01.dbf /p03/oradata/OSID/sysaux01.dbf /p03/oradata/OSID/users01.dbf /p03/oradata/OSID/example01.dbf /p03/oradata/OSID/data01.dbf /p02/oradata/OSID/index01.dbf /p01/oradata/OSID/control01.ctl /p02/oradata/OSID/control02.ctl /p03/oradata/OSID/control03.ctl /s01/oracle/product/10.2.0/dbs/initOSID.ora Requiring a command line something like this: rsync -a srchost:/ / --files-from=dbf-list and dbf-list would contain this: p02/oradata/OSID/redo01.log p03/oradata/OSID/redo02.log p04/oradata/OSID/redo03.log p01/oradata/OSID/system01.dbf p04/oradata/OSID/undotbs01.dbf p03/oradata/OSID/sysaux01.dbf p03/oradata/OSID/users01.dbf p03/oradata/OSID/example01.dbf p03/oradata/OSID/data01.dbf p02/oradata/OSID/index01.dbf p01/oradata/OSID/control01.ctl p02/oradata/OSID/control02.ctl p03/oradata/OSID/control03.ctl s01/oracle/product/10.2.0/dbs/initOSID.ora Do you see any problems with this? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync with bwlimit
Hi, Here's one I sometimes use which limits the bandwidth to a maximum of 64k rsync -avz --bwlimit=64 10.0.10.130::Backup /backupdir -- Reuben Pearse Principal Technical Consultant [EMAIL PROTECTED] T +44 121 428 4440 F +44 121 428 3454 PSL The Old Sewing Factory 5a Greenfield Road Harborne Birmingham B17 0ED To find out more about our services please visit http://www.pslconnect.com redlotus72 [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 28/02/2006 13:03 To [EMAIL PROTECTED] cc Subject Rsync with bwlimit Hi, Any one give me some sample code with bwlimit option. what's is the std/ recommended value for this? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- CONFIDENTIALITY NOTICE The information contained in this e-mail is intended only for the confidential use of the above named recipient. If you are not the intended recipient or person responsible for delivering it to the intended recipient, you have received this communication in error and must not distribute or copy it. Please accept the sender's apologies, notify the sender immediately by return e-mail and delete this communication. Thank you.-- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync help needed...
On Fri, 2006-02-24 at 11:08 -0800, lsk wrote:
I have been using the following syntax..
rsync -cvz /d01/app/testfile1.dbf tarser:/t01/app/testfile1.dbf
but I would change to the one below and test a 40 GB transfer and see the
results...
rsync -zv --no-whole-file --stats /d01/app/testfile2.dbf
tarser:/t01/app/testfile2.dbf
Yes, make that change! You don't need --no-whole-file; it's the default
because you're doing a remote transfer. Leaving the old file on the
receiver and omitting --checksum has already brought down the transfer
time significantly in your earlier test; I bet --inplace will cut the
time another 20-40% or so.
--
Matt McCutchen
[EMAIL PROTECTED]
http://hashproduct.metaesthetics.net/
lsk
Matt I have strange results to report I transferred 300 oracle datafiles of
total 30 GB in size. Using the option rsync -zv --no-whole-file --stats
it took 1:15 min and using rsync -cvz options earlier had took 1:25 min so
there wasn't much time savings I thought it would be a big difference since
the files are there and only little is modified (header info) + additional
datafiles.
{{ Other info is I am running rsync version 2.6.5 protocol version 29 on
source and rsync version 2.5.2 protocol version 26 on destination server
}}
Also one more thing from the stats rsync has transferred few files more than
once Number of files transferred: 2 Why is this so ?
Number of files: 1
Number of files transferred: 2
Total file size: 167780352 bytes
Total transferred file size: 335560704 bytes
Literal data: 67010560 bytes
Matched data: 268550144 bytes
File list size: 42
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 184089
Total bytes received: 254026
sent 184089 bytes received 254026 bytes 2131.95 bytes/sec
total size is 167780352 speedup is 382.96
Thanks,
lsk.
--
View this message in context:
http://www.nabble.com/Rsync-help-needed...-t1170765.html#a3164851
Sent from the Samba - rsync forum at Nabble.com.
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync help needed...
On Fri, 2006-02-24 at 11:08 -0800, lsk wrote:
I have been using the following syntax..
rsync -cvz /d01/app/testfile1.dbf tarser:/t01/app/testfile1.dbf
but I would change to the one below and test a 40 GB transfer and see the
results...
rsync -zv --no-whole-file --stats /d01/app/testfile2.dbf
tarser:/t01/app/testfile2.dbf
Yes, make that change! You don't need --no-whole-file; it's the default
because you're doing a remote transfer. Leaving the old file on the
receiver and omitting --checksum has already brought down the transfer
time significantly in your earlier test; I bet --inplace will cut the
time another 20-40% or so.
--
Matt McCutchen
[EMAIL PROTECTED]
http://hashproduct.metaesthetics.net/
lsk
Matt I have strange results to report I transferred 300 oracle datafiles of
total 30 GB in size. Using the option rsync -zv --no-whole-file --stats
it took 1:15 min and using rsync -cvz options earlier had took 1:25 min so
there wasn't much time savings I thought it would be a big difference since
the files are there and only little is modified (header info) + additional
datafiles.
{{ Other info is I am running rsync version 2.6.5 protocol version 29 on
source and rsync version 2.5.2 protocol version 26 on destination server
}}
Also one more thing from the stats rsync has transferred few files more than
once Number of files transferred: 2 Why is this so ?
Number of files: 1
Number of files transferred: 2
Total file size: 167780352 bytes
Total transferred file size: 335560704 bytes
Literal data: 67010560 bytes
Matched data: 268550144 bytes
File list size: 42
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 184089
Total bytes received: 254026
sent 184089 bytes received 254026 bytes 2131.95 bytes/sec
total size is 167780352 speedup is 382.96
Thanks,
lsk.
--
View this message in context:
http://www.nabble.com/Rsync-help-needed...-t1170765.html#a3164850
Sent from the Samba - rsync forum at Nabble.com.
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Crazy excludes
On Tue, Feb 28, 2006 at 12:54:53PM +0100, Matias Surdi wrote: #NEXT COMMAND, IS JUST AN ECHO OF THE COMMAND THAT WILL BE RUN IN NEXT STEP. - rsync -a -v --delete --timeout 120 --exclude='/msurdi/*' localhost:/home/ /usr/backups/gaia.0/ Since you can see the single quotes in the output of that echo, that means that you're excluding files that start with a single quote (i.e. nothing). I assume you've over-escaped your single quotes. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Crazy excludes
Wayne Davison escribió: On Tue, Feb 28, 2006 at 12:54:53PM +0100, Matias Surdi wrote: #NEXT COMMAND, IS JUST AN ECHO OF THE COMMAND THAT WILL BE RUN IN NEXT STEP. - rsync -a -v --delete --timeout 120 --exclude='/msurdi/*' localhost:/home/ /usr/backups/gaia.0/ Since you can see the single quotes in the output of that echo, that means that you're excluding files that start with a single quote (i.e. nothing). I assume you've over-escaped your single quotes. ..wayne.. Thanks Wayne.. that was the problem. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Copying many sources to different places inside a destination
On Mon, 2006-02-27 at 23:18 -0800, Wayne Davison wrote:
A similar symlink solution could be done on the source side using the
new --copy-dirlinks option, but only if the nesting of directories
allowed it to happen (which does not seem to be the case in your
example).
That would also work if I made symlinks from main-web-content to the
other sources, which isn't a problem. However, I might have other
dirlinks in the sources that I don't want to follow. I found a way to
make rsync follow only certain dirlinks, taking advantage of two lucky
behaviors: (1) lstat follows a dirlink specified with a trailing slash,
and (2) rsync filters out duplicates in the file list, favoring
directories over non-directories. If you like, stop and see if you can
figure out the technique from those hints.
The solution is: use --relative, don't use --copy-dirlinks, and list the
dirlinks to follow as source arguments with trailing slashes (along with
the main source). rsync recurses inside the main source and does not
follow dirlinks: instead it generates symlink flist entries for them.
Then it recurses inside the target of each specified dirlink and
generates a directory flist entry for the dirlink (and entries for the
files inside). During flist filtering, rsync drops the symlink entries
for the specified dirlinks.
The ability for a sending rsync to give files file-list names
independent of their real locations in the filesystem would still be
nice even though I can solve my problem without it. The way to go about
this would probably be to replace file_struct::dir.root with a pointer
to an instance of this structure:
struct source_arg {
char *fs_prefix; /* analogue of root */
char *flist_prefix;
};
Then, when the sender sends a file-list entry to the receiver, it could
attach the flist_prefix to the beginning of the dirname. The sender
would also have to create and fill source_arg structures while building
the file list. The same system that was to be used to allocate id_pairs
could be used for source_args. This doesn't sound too hard; I might try
to implement it.
--
Matt McCutchen
[EMAIL PROTECTED]
http://hashproduct.metaesthetics.net/
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync help needed...
On Tue, 2006-02-28 at 09:20 -0500, Linus Hicks wrote: rsync -a srchost:/ / --files-from=dbf-list and dbf-list would contain this: p02/oradata/OSID/redo01.log p03/oradata/OSID/redo02.log p04/oradata/OSID/redo03.log p01/oradata/OSID/system01.dbf p04/oradata/OSID/undotbs01.dbf p03/oradata/OSID/sysaux01.dbf p03/oradata/OSID/users01.dbf p03/oradata/OSID/example01.dbf p03/oradata/OSID/data01.dbf p02/oradata/OSID/index01.dbf p01/oradata/OSID/control01.ctl p02/oradata/OSID/control02.ctl p03/oradata/OSID/control03.ctl s01/oracle/product/10.2.0/dbs/initOSID.ora Do you see any problems with this? That should work fine; that is exactly how --files-from is typically used. Note that rsync looks for dbf-list on the computer on which you are starting rsync; if dbf-list is on the other computer, use --files-from=:dbf-list to indicate this. -- Matt McCutchen [EMAIL PROTECTED] http://hashproduct.metaesthetics.net/ -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync help needed...
On Tue, 2006-02-28 at 07:43 -0800, lsk wrote:
Matt I have strange results to report I transferred 300 oracle datafiles of
total 30 GB in size. Using the option rsync -zv --no-whole-file --stats
it took 1:15 min and using rsync -cvz options earlier had took 1:25 min so
there wasn't much time savings I thought it would be a big difference since
the files are there and only little is modified (header info) + additional
datafiles.
Hm. I guess the checksumming was only taking 10 minutes, which means
you must have a fast disk! Adding --inplace should give a bigger
improvement.
{{ Other info is I am running rsync version 2.6.5 protocol version 29 on
source and rsync version 2.5.2 protocol version 26 on destination server
}}
You might want to upgrade the destination server.
Also one more thing from the stats rsync has transferred few files more than
once Number of files transferred: 2 Why is this so ?
Number of files: 1
Number of files transferred: 2
Total file size: 167780352 bytes
Total transferred file size: 335560704 bytes
Literal data: 67010560 bytes
Matched data: 268550144 bytes
File list size: 42
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 184089
Total bytes received: 254026
sent 184089 bytes received 254026 bytes 2131.95 bytes/sec
total size is 167780352 speedup is 382.96
Once in a while, a block hash collision will occur in rsync's
incremental transfer algorithm, causing the receiver to falsely decide
it already has part of the file. When this happens, the after-transfer
checksum fails, a warning is printed, and the transfer starts over; this
second transfer is being counted in the stats.
It looks like you're invoking rsync once per file. If you create a file
listing all the filenames and invoke rsync once overall with
--files-from, you can reduce the overhead of SSH-ing and setting up the
rsync process triangle.
--
Matt McCutchen
[EMAIL PROTECTED]
http://hashproduct.metaesthetics.net/
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync help needed...
On Tue, Feb 28, 2006 at 07:45:20AM -0800, lsk wrote: Matt I have strange results to report I transferred 300 oracle datafiles of total 30 GB in size. Using the option rsync -zv --no-whole-file --stats it took 1:15 min and using rsync -cvz options earlier had took 1:25 min so there wasn't much time savings I thought it would be a big difference since the files are there and only little is modified (header info) + additional datafiles. My guess is that the number of files and size dominates; since even for a small change, rsync has to write the entire file. You could try the (unsupported, not included?) --in-place feature, available in a patch somewhere. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Tue Feb 28 21:48:45 2006 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv22137 Modified Files: NEWS Log Message: One more bugfix. Revisions: NEWS1.377 = 1.378 http://www.samba.org/cgi-bin/cvsweb/rsync/NEWS?r1=1.377r2=1.378 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/patches
Date: Tue Feb 28 21:49:42 2006 Author: wayned Update of /data/cvs/rsync/patches In directory dp.samba.org:/tmp/cvs-serv22231 Modified Files: dynamic_hash.diff Log Message: Most of the patch got applied to the trunk, so what remains is just the bit that makes the hash table dynamically sized. Revisions: dynamic_hash.diff 1.7 = 1.8 http://www.samba.org/cgi-bin/cvsweb/rsync/patches/dynamic_hash.diff?r1=1.7r2=1.8 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Wed Mar 1 00:59:22 2006 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv21494 Modified Files: NEWS Log Message: Improved a couple sentences. Revisions: NEWS1.378 = 1.379 http://www.samba.org/cgi-bin/cvsweb/rsync/NEWS?r1=1.378r2=1.379 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Wed Mar 1 01:05:09 2006 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv25766 Modified Files: rsync.1 Log Message: Regenerated. Revisions: rsync.1 1.372 = 1.373 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.1?r1=1.372r2=1.373 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Wed Mar 1 01:04:57 2006 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv25461 Modified Files: rsync.yo Log Message: A couple more improvements for --list-only. Revisions: rsync.yo1.360 = 1.361 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.yo?r1=1.360r2=1.361 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
