Re: Rsync 4TB datafiles...?
On Wed, 17 May 2006, lsk [EMAIL PROTECTED] wrote: So this shows that RSYNC performs faster when there is no datafile is present. For Oracle datafiles the diff algorithm doesn't work even with the patch. Literally it should have taken less than 12 min to just compare and update the header changes but it took more than 12 min when the file is already present at destination(target). Are you using the --inplace option? If the file doesn't exist, the entire file is sent, yes, but there's only a serial write operation needed at the destination. If the file *does* exist, rsync must read the existing file in its entirety while *also* writing out the new file. The dance that the disk R/W head might be doing in this case might slow things down, too. I am not going to give up I am going to try again and see... OK. :) John -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
TEST RESULTS after applying the patch Dynamic_hash.diff with new rsync version 2.6.8. -- The patch was installed on the source server and I tested with a 2 GB file transfer with following options and these are the timings... 1. No DESTINATION File - 2 GB - 12 min 2. With DESTINATION File nothing changed - 2 GB - 21 min 3. With DESTINATION File header information changed - 17 min The first 2 were tested with rysnc -zv and the last one with rsync -zv --no-whole-file So this shows that RSYNC performs faster when there is no datafile is present. For Oracle datafiles the diff algorithm doesn't work even with the patch. Literally it should have taken less than 12 min to just compare and update the header changes but it took more than 12 min when the file is already present at destination(target). I am not going to give up I am going to try again and see... Thanks, lsk. -- View this message in context: http://www.nabble.com/Rsync-4TB-datafiles...--t1318624.html#a4429359 Sent from the Samba - rsync forum at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
Also I use the rsync version rsync version 2.6.5 protocol version 29 does this version include this patch dynamic_hash.diff or do we need to install it seperately. Sorry. You will need to get the 2.6.7 sources, and then apply the patch yourself and compile rsync. Please do report back here your results. This patch is a result of a lot of theoretical work, but we never got any actual feedback on it. Shachar -- ***Hello Shachar...is 2.6.7 is the latest version of rsync. I could see in the http download site it says rsync-2.6.8.tar.gz. Should I get this version 2.6.8 + the patch dynamic_hash.diff. Also I am planning to install in only the sending machine...and first try out. Thanks for your feedback. lsk. -- View this message in context: http://www.nabble.com/Rsync-4TB-datafiles...--t1318624.html#a4180739 Sent from the Samba - rsync forum at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
lsk wrote: Hello Shachar...is 2.6.7 is the latest version of rsync. I could see in the http download site it says rsync-2.6.8.tar.gz. Should I get this version 2.6.8 + the patch dynamic_hash.diff. Yes. In the over a month that passed since the email I sent a new version of rsync was released :-) Dynamic_hash.diff is available in that one too. Also I am planning to install in only the sending machine...and first try out. Should work. Thanks for your feedback. lsk. Shachar -- Shachar Shemesh Lingnu Open Source Consulting ltd. Have you backed up today's work? http://www.lingnu.com/backup.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
lsk wrote: Hello Shachar...is 2.6.7 is the latest version of rsync. I could see in the http download site it says rsync-2.6.8.tar.gz. Should I get this version 2.6.8 + the patch dynamic_hash.diff. Yes. In the over a month that passed since the email I sent a new version of rsync was released :-) Dynamic_hash.diff is available in that one too. //***Shachar is Dynamic_hash.diff in built in 2.6.8 code version or still I need to apply that as a seperate patch after I install rsync 2.6.8 ? Thanks, lsk. -- View this message in context: http://www.nabble.com/Rsync-4TB-datafiles...--t1318624.html#a4198608 Sent from the Samba - rsync forum at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
On Tue, 2006-05-02 at 13:06 -0700, lsk wrote: //***Shachar is Dynamic_hash.diff in built in 2.6.8 code version or still I need to apply that as a seperate patch after I install rsync 2.6.8 ? You need to apply it as a separate patch _before you build_ rsync 2.6.8. Apply the patch by running patch -p1 patches/dynamic_hash.diff in the source tree. Then configure, make, and install. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
On Wed 22 Mar 2006, Linus Hicks wrote: Paul Slootman wrote: I'd recommend doing --inplace, as chances are that data won't move within a file with oracle data files (so it's not useful to try to find moved data), and copying the 4TB to temp. files every time could become a big timewaster. Also the -t option could be handy, not all files change all the time IIRC. The above remark about not being useful to try to find moved data provoked an idea. But my understanding of --inplace is apparently different from yours. I thought --inplace only meant that the destination file would be directly overwritten, not that it would turn off any of the optimizations for finding moved data. I go on what's in the manpage: --inplace This causes rsync not to create a new copy of the file and then move it into place. Instead rsync will overwrite the existing file, meaning that the rsync algorithm can't accomplish the full amount of network reduction it might be able to otherwise (since it does not yet try to sort data matches). One exception to this is if you combine the option with --backup, since rsync is smart enough to use the backup file as the basis file for the transfer. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
lsk wrote: But I have tried various options including --inplace,--no-whole-file etc., for last few weeks but all the results show me removing the destination server oracle datafiles and after that doing an rsync -vz from source is faster than copying(rsyncing) over the old files that are present in destination. Please do try applying the patch in patches/dynamic_hash.diff to both sides (well, it's probably only necessary for the sending machine, but no matter) and making this check again. This patch is meant to address precisely your predicament. Shachar -- /// lsk Hi which issue does this patch address ? it is for -- inplace or for ,--no-whole-file transfer of oracle datafiles. Where can I get this patch ? Thanks, lsk. -- View this message in context: http://www.nabble.com/Rsync-4TB-datafiles...--t1318624.html#a3534475 Sent from the Samba - rsync forum at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
Also I use the rsync version rsync version 2.6.5 protocol version 29 does this version include this patch dynamic_hash.diff or do we need to install it seperately. At destination server I use rsync version 2.6.6 protocol version 29 anyhow u said that doesn't matter. Thanks, lsk. -- View this message in context: http://www.nabble.com/Rsync-4TB-datafiles...--t1318624.html#a3534654 Sent from the Samba - rsync forum at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
Paul Slootman wrote: On Tue 21 Mar 2006, lsk wrote: I don't know how it would work if we do rsync with the files--from option ? I'm not sure how rsync behaves when confronted with a network problem during a session, so I won't give an answer to that. However, doing individual files sounds reasonable, so make it a loop: dbf-list while read filename; do rsync -vz $filename destser:$filename done Also rsync gurus would you suggest which is the fastest way to trasfer this 4 TB data ? Any suggestions...would be of great help. I'd recommend doing --inplace, as chances are that data won't move within a file with oracle data files (so it's not useful to try to find moved data), and copying the 4TB to temp. files every time could become a big timewaster. Also the -t option could be handy, not all files change all the time IIRC. The above remark about not being useful to try to find moved data provoked an idea. But my understanding of --inplace is apparently different from yours. I thought --inplace only meant that the destination file would be directly overwritten, not that it would turn off any of the optimizations for finding moved data. It would be useful (I think) on a fast network to be able to turn off those optimizations, and only compare blocks located at the same offset in source and destination. If that is not how --inplace works, I wonder if that would be a performance win. Linus -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
lsk wrote: Also I use the rsync version rsync version 2.6.5 protocol version 29 does this version include this patch dynamic_hash.diff or do we need to install it seperately. Sorry. You will need to get the 2.6.7 sources, and then apply the patch yourself and compile rsync. Please do report back here your results. This patch is a result of a lot of theoretical work, but we never got any actual feedback on it. Shachar -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync 4TB datafiles...?
I need to rsync 4 TB datafiles to remote server and clone to a new oracle database..I have about 40 drives that contains this 4 TB data. I would like to do rsync from a directory level by using --files-from=FILE option. But the problem is what will happen if the network connection fails the whole rsync will fail right. rsync -a srchost:/ / --files-from=dbf-list and dbf-list would contain this: p02/oradata/OSID/redo01.log p03/oradata/OSID/redo02.log p04/oradata/OSID/redo03.log p01/oradata/OSID/system01.dbf p04/oradata/OSID/undotbs01.dbf p03/oradata/OSID/sysaux01.dbf p03/oradata/OSID/users01.dbf p03/oradata/OSID/example01.dbf p03/oradata/OSID/data01.dbf p02/oradata/OSID/index01.dbf p01/oradata/OSID/control01.ctl p02/oradata/OSID/control02.ctl p03/oradata/OSID/control03.ctl s01/oracle/product/10.2.0/dbs/initOSID.ora One advantage that I have if do rsync file by file is if the network connection fails for a quite sometime and comes back alive the rsync will continue from the next file in the list while it will fail on the other files when connection is lost. rsync -vz /p02/oradata/OSID/redo01.log destser:/d02/oradata/OSID/redo01.log rsync -vz /p03/oradata/OSID/redo02.log destser:/d03/oradata/OSID/redo02.log rsync -vz /p04/oradata/OSID/redo03.log destser:/d04/oradata/OSID/redo03.log rsync -vz /p01/oradata/OSID/system01.dbf destser:/d01/oradata/OSID/system01.dbf rsync -vz /p04/oradata/OSID/undotbs01.dbf destser:/d04/oradata/OSID/undotbs01.dbf rsync -vz /p03/oradata/OSID/sysaux01.dbf destser:/d03/oradata/OSID/sysaux01.dbf rsync -vz /p03/oradata/OSID/users01.dbf destser:/d03/oradata/OSID/users01.dbf rsync -vz /p03/oradata/OSID/example01.dbf destser:/d03/oradata/OSID/example01.dbf rsync -vz /p03/oradata/OSID/data01.dbf destser:/d03/oradata/OSID/data01.dbf rsync -vz /p02/oradata/OSID/index01.dbf destser:/d02/oradata/OSID/index01.dbf rsync -vz /p01/oradata/OSID/control01.ctl destser:/d01/oradata/OSID/control01.ctl rsync -vz /p02/oradata/OSID/control02.ctl destser:/d02/oradata/OSID/control02.ctl rsync -vz /p03/oradata/OSID/control03.ctl destser:/d03/oradata/OSID/control03.ctl rsync -vz /s01/oracle/product/10.2.0/dbs/initOSID.ora destser:/d01/oracle/droduct/10.2.0/dbs/initOSID.ora I don't know how it would work if we do rsync with the files--from option ? Also rsync gurus would you suggest which is the fastest way to trasfer this 4 TB data ? Any suggestions...would be of great help. Thanks, lsk. -- View this message in context: http://www.nabble.com/Rsync-4TB-datafiles...--t1318624.html#a3516542 Sent from the Samba - rsync forum at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
On Tue 21 Mar 2006, lsk wrote: I don't know how it would work if we do rsync with the files--from option ? I'm not sure how rsync behaves when confronted with a network problem during a session, so I won't give an answer to that. However, doing individual files sounds reasonable, so make it a loop: dbf-list while read filename; do rsync -vz $filename destser:$filename done Also rsync gurus would you suggest which is the fastest way to trasfer this 4 TB data ? Any suggestions...would be of great help. I'd recommend doing --inplace, as chances are that data won't move within a file with oracle data files (so it's not useful to try to find moved data), and copying the 4TB to temp. files every time could become a big timewaster. Also the -t option could be handy, not all files change all the time IIRC. As always, get the latest version. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
Also rsync gurus would you suggest which is the fastest way to trasfer this 4 TB data ? Any suggestions...would be of great help. I'd recommend doing --inplace, as chances are that data won't move within a file with oracle data files (so it's not useful to try to find moved data), and copying the 4TB to temp. files every time could become a big timewaster. Also the -t option could be handy, not all files change all the time IIRC. ***lsk I can't use -t because the oracle datafiles header information are constanly updated so it will definetely change. But I have tried various options including --inplace,--no-whole-file etc., for last few weeks but all the results show me removing the destination server oracle datafiles and after that doing an rsync -vz from source is faster than copying(rsyncing) over the old files that are present in destination. I put list of rsync file commands in a batch file and spawn them so that they run in background. Thanks, lsk. -- View this message in context: http://www.nabble.com/Rsync-4TB-datafiles...--t1318624.html#a3518597 Sent from the Samba - rsync forum at Nabble.com. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 4TB datafiles...?
lsk wrote: But I have tried various options including --inplace,--no-whole-file etc., for last few weeks but all the results show me removing the destination server oracle datafiles and after that doing an rsync -vz from source is faster than copying(rsyncing) over the old files that are present in destination. Please do try applying the patch in patches/dynamic_hash.diff to both sides (well, it's probably only necessary for the sending machine, but no matter) and making this check again. This patch is meant to address precisely your predicament. Shachar -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
