Re: rsnyc over ssh through scripting and cron...
Hi, On Sunday 29 January 2006 19:36, Quinn Snyder wrote: I have two machines (approx 50 miles apart) that I need to create a co-located backup solution for. One machine will serve as a central fileserver for an office, and every week a backup of that directory will be sent through the internet to a box I have here locally. I am aware of rsync being able to run over SSH, however I need this to be totally automated (SSH opening, passwords sent and synchronized, files copied, and connections broken). If I can create the script, I can place the script in the cron.weekly folder, and execute it from there using cron. I just need to know where to start, and how I'm going to get the password automation (any links or anything.I don't need to be walked through it, I just need a starting point, though any help is appreciated). You need a pre-shared key without passphrase: http://www.google.com/search?q=openssh+public+key to tunnel rsync thru SSH without user interaction. Yours, -- René Rebe - Rubensstr. 64 - 12157 Berlin (Europe / Germany) http://www.exactcode.de | http://www.t2-project.org +49 (0)30 255 897 45 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Shell Script Does not complete if rsync returns Code 24
Hi, On Wednesday 11 January 2006 04:28, John Gallagher wrote: The problem is that I have no clue what to do with this and or how to make it work with my script. If you want hide errors, remove the -e from your sh invocation or add || true at the end of the rsync invocation. Though I would not do either in a backup script. Better would be to just ignore some of rsync's return codes. Regards, -- René Rebe - Rubensstr. 64 - 12157 Berlin (Europe / Germany) http://www.exactcode.de | http://www.t2-project.org +49 (0)30 255 897 45 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: performance with 50GB files
Hi,
On Wednesday 11 January 2006 00:31, Wayne Davison wrote:
So, perhaps the size of the sending file should be factored into the
calculation in order to set a minimum acceptable block size. This would
be easy, because the generator already knows the size of both files at
the time that it performs this calculation (or else it wouldn't have
been able to figure out if the file was up-to-date or not).
So, here is some hackery to scale the block size from 700 bytes for a near
zero sized file to 5MB for a 50GB file and linear increasing using the bigger
size
of the the two available sources.
But I fear you want to keep something like the old, rather overly complex IMHO
block size algorithm but tweak some factor in it ...
--- rsync-2.6.6/generator.c 2005-07-28 21:06:03.0 +0200
+++ rsync-2.6.6-fixed/generator.c 2006-01-11 12:34:01.0 +0100
@@ -405,25 +405,11 @@
if (block_size)
blength = block_size;
- else if (len = BLOCK_SIZE * BLOCK_SIZE)
- blength = BLOCK_SIZE;
else {
- int32 c;
- int64 l;
- int cnt;
- for (c = 1, l = len, cnt = 0; l = 2; c = 1, cnt++) {}
- if (cnt = 31 || c = MAX_BLOCK_SIZE)
+ blength = len / 1 + BLOCK_SIZE;
+ blength -= blength % 8; /* round to multiple of 8 */
+ if (blength = MAX_BLOCK_SIZE)
blength = MAX_BLOCK_SIZE;
- else {
- blength = 0;
- do {
- blength |= c;
- if (len (int64)blength * blength)
- blength = ~c;
- c = 1;
- } while (c = 8); /* round to multiple of 8 */
- blength = MAX(blength, BLOCK_SIZE);
- }
}
if (protocol_version 27) {
@@ -462,14 +448,14 @@
*
* Generate approximately one checksum every block_len bytes.
*/
-static void generate_and_send_sums(int fd, OFF_T len, int f_out, int f_copy)
+static void generate_and_send_sums(int fd, OFF_T len, OFF_T len2, int f_out,
int f_copy)
{
int32 i;
struct map_struct *mapbuf;
struct sum_struct sum;
OFF_T offset = 0;
- sum_sizes_sqroot(sum, len);
+ sum_sizes_sqroot(sum, MAX(len, len2));
if (len 0)
mapbuf = map_file(fd, len, MAX_MAP_SIZE, sum.blength);
@@ -1123,7 +1109,7 @@
return;
}
- generate_and_send_sums(fd, st.st_size, f_out, f_copy);
+ generate_and_send_sums(fd, st.st_size, file-length, f_out, f_copy);
if (f_copy = 0) {
close(f_copy);
--
René Rebe - Rubensstr. 64 - 12157 Berlin (Europe / Germany)
http://www.exactcode.de | http://www.t2-project.org
+49 (0)30 255 897 45
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: performance with 50GB files
Hi, in reply to my previous post, I can reproduce the issue locally here. I produced a 50146750688 bytes /home/test.dat out of cat'ing a lot of data files together (needed some input data ...). The initial rsync take over an hour saturating the 100mbit ethernet. I then used shred on the first GB of the file and cat'ed some more data to the end and rerun rsync: rsync -arvPe ssh 192.168.2.45:/home/test.dat test.dat the sending Athlon XP 2500+ is saturated while the receiver gets: 619708416 1%1.03MB/s 13:02:58 of course the dual cpu ppc64 receiver is idling waiting for any data to arrive. The sender: 22952 root 18 0 12988 11m 624 D 95.0 2.2 5:24.75 rsync Oprofile shows: samples %symbol name 9273739 87.1946 match_sums 6339105.9602 map_ptr 4598174.3233 mdfour64 2179742.0495 copy64 32467 0.3053 mdfour_update 11310 0.1063 get_checksum2 2206 0.0207 writefd_unbuffered 871 0.0082 sum_update 638 0.0060 writefd 522 0.0049 mplex_write 306 0.0029 compare_targets 256 0.0024 io_flush 253 0.0024 matched 241 0.0023 send_token 229 0.0022 msg_list_push 198 0.0019 mdfour_tail 128 0.0012 write_int 122 0.0011 readfd_unbuffered 105 9.9e-04 readfd 94 8.8e-04 send_files 69 6.5e-04 mdfour_begin 63 5.9e-04 mdfour_result 37 3.5e-04 write_buf 33 3.1e-04 .plt 25 2.4e-04 copy4 25 2.4e-04 read_int 19 1.8e-04 read_buf 11 1.0e-04 read_timeout 10 9.4e-05 get_checksum1 19.4e-06 _fini 19.4e-06 clean_flist 19.4e-06 deflate_fast 19.4e-06 parse_arguments So far ... I continue to analyze the issue, maybe some rsync developer already comes to a conclusionn while I start reading thru the source. Best regards, On Monday 09 January 2006 23:38, René Rebe wrote: today we had a performance issue transfering a big amount of data where one file was over 50GB. Rsync was tunneled over SSH and we expected the data to be synced within hours. However after over 10 hours the data is still not synced ... The sending box has rsync running with 60-80 % CPU load (2GHz Pentium 4) while the receiver is nearly idle. So far I had no acces to the poblematic setup but I will have to analyze this soon. I would like to ask beforehand if there are known performance hits syncing such huge files? -- René Rebe - Rubensstr. 64 - 12157 Berlin (Europe / Germany) http://www.exactcode.de | http://www.t2-project.org +49 (0)30 255 897 45 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: performance with 50GB files
Hi, On Tuesday 10 January 2006 21:47, Wayne Davison wrote: On Tue, Jan 10, 2006 at 09:02:14PM +0100, Ren? Rebe wrote: So far just increasing the block-size significantly (10-20MB) bumps the speed by magnitudes into useful regions. That's good. For some reason I was thinking that the block size was nearly maxxed out for a 50GB file, but I can see that the current algorithm for selecting the block size ends up nowhere near the maximum block-size limit. Yes, I think this could be fixed up quite easily. Also I found the current code does decide from the receiving-side file what blocksize to use. Now with the case where a receiver file is way smaller than the reference one, it would pick a rather tiny blocksize due to the available tiny file and then take nearly a day due to the tiny blocksize. The same applies when the sync was aborted and restarted. But getting the sent size to the code selecting the blocksize is not a too tiny change. Regards, -- René Rebe - Rubensstr. 64 - 12157 Berlin (Europe / Germany) http://www.exactcode.de | http://www.t2-project.org +49 (0)30 255 897 45 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
performance with 50GB files
Hi all, today we had a performance issue transfering a big amount of data where one file was over 50GB. Rsync was tunneled over SSH and we expected the data to be synced within hours. However after over 10 hours the data is still not synced ... The sending box has rsync running with 60-80 % CPU load (2GHz Pentium 4) while the receiver is nearly idle. So far I had no acces to the poblematic setup but I will have to analyze this soon. I would like to ask beforehand if there are known performance hits syncing such huge files? Sincerely, René Rebe -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
