Re: rsync speedup - how ?
On Fri, 2009-08-07 at 18:44 +0200, devz...@web.de wrote: devz...@web.de wrote: so, instead of 500M i would transfer 100GB over the network. that`s no option. I don't see how you came up with such numbers. If files change completely then I don't see why you would transfer more (or less) over the network. The difference that I'm thinking of is that by not using the rsync algorithm then you're substantially reducing the number of disk I/Os. let me explain: all files are HUGE datafiles and they are of constant size. they are harddisk-images and the contents being changed inside, i.e. specific blocks in the files being accessed and rewritten. so, the question is: is rsync rolling checksum algorithm the perfect (i.e. fastest) algorithm to match changed blocks at fixed locations between source and destination files ? i`m not sure because i have no in depth knowledge of the mathematical background in rsync algorithm. i assume: no - but it`s only a guess... I really don't think it's a good idea to sync large data files in use, which is modified frequently, e.g. SQL database, VMware image file. As rsync do NOT have the algorithm to keep those frequently modified data file sync with the source file. And this will course data file corrupted. If I'm wrong, please correct me. Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How to get first level folder names with rsync commands?
Dear List, I set up a NetBackup destination on my server. And I backup test1 to NetBackup/test1, test2 to NetBackup/test2. Now I wanna know how many folders that have been backuped. It should be 2 folders, with the name of test1 and test2. But I don't know if there is any way to do so? Any advice and help is well appreciated. Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How to first level folder names with rsync commands?
Dear List, I set up a NetBackup destination on my server. And I backup test1 to NetBackup/test1, test2 to NetBackup/test2. Now I wanna know how many folders that have been backuped. It should be 2 folders, with the name of test1 and test2. But I don't know if there is any way to do so? Any advice and help is well appreciated. Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: How to get first level folder names with rsync commands?
Well, what I mean is running some rsync command, and get a list something like. $ rsync options destination List: test1 test2 Is it possible with rsync options? On Wed, 2009-07-01 at 00:23 -0500, Tony Abernethy wrote: Something like ls -d NetBackup/* | wc should be informative. (assuming something unixy, of course -- cygwin stuff might work) -Original Message- From: rsync-bounces+tony=servacorp@lists.samba.org [mailto:rsync-bounces+tony=servacorp@lists.samba.org] On Behalf Of Daniel.Li Sent: Wednesday, July 01, 2009 12:17 AM To: Rsync Subject: How to get first level folder names with rsync commands? Dear List, I set up a NetBackup destination on my server. And I backup test1 to NetBackup/test1, test2 to NetBackup/test2. Now I wanna know how many folders that have been backuped. It should be 2 folders, with the name of test1 and test2. But I don't know if there is any way to do so? Any advice and help is well appreciated. Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-acv option, what will be the default delete option?
Dear List, If I use -acv for rsync options, what will I choose as default delete option? --deletedelete extraneous files from destination dirs --delete-before receiver deletes before transfer, not during --delete-during receiver deletes during transfer (default) --delete-delay find deletions during, delete after --delete-after receiver deletes after transfer, not during --delete-excluded also delete excluded files from destination dirs Thanks in advance! -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Weird behavior in receive_data function
Dear List,
I'm trying to get diff/removed data and it's offset out. So I write a
functions in receive_data. When I run backup, I found there is a weird
behavior which I don't understand.
i = recv_token(f_in, data) will receive (i = -1, offset2 = 0) some
where in the middle of the transfer procedure. That's to say, it's going
to transfer the first data block from sender, but I have been
transferring almost half of the file. I noticed it was 92nd block
(sum.blength = 700) .
Hope I have a clear explanation. I'm really confused now. Any advice and
suggestion is well appreciated.
Below is my code segment to generate file for test.
My Procedure (file 1 and file 2 have the same name, just binary
difference):
Step 1: backup File 1
Step 2: backup Fil 2
--
Daniel Li
File 1:
This is the first time backup file.
count = 1;
nSize = 1;
nBlock = 256;
nInterval = 4;
nAdjust = 256;
printf(\nstart to build origin files!\n);
ch = 0;
for(l = 0; l count; l++)
{
memset(buffer,0, 256);
sprintf(buffer,media-test-3MB-file/origin/test/test%02d.dat,l);
printf(writting %s \n,buffer);
fd = open(buffer, O_RDWR|O_CREAT|O_EXCL,
S_IRUSR|S_IWUSR);
for(lll = 0; lll nSize; lll++)
{
for(mmm = 0; mmm nAdjust; mmm++)
{
for(nnn = 0; nnn nBlock; nnn++)
{
write_fhead_char(fd, ch, 1);
}
ch++;
}
}
}
printf(origin files done!\n);
File 2:
This is the second time backup. when transferring this file, I met the
trouble.
printf(\nstart to build added data files!\n);
ch = 0;
for(l = 0; l count; l++)
{
memset(buffer,0, 256);
sprintf(buffer,media-test-3MB-file/add/test/test%02d.dat,l);
printf(writting %s \n,buffer);
fd = open(buffer, O_RDWR|O_CREAT|O_EXCL,
S_IRUSR|S_IWUSR);
for(lll = 0; lll nSize + 1; lll++)
{
for(mmm = 0; mmm nAdjust; mmm++)
{
if(mmm % nInterval == 0)
{
for(nnn = 0; nnn nBlock;
nnn++)
{
write_fhead_char(fd,
ch, 1);
}
}
for(nnn = 0; nnn nBlock; nnn++)
{
write_fhead_char(fd, ch, 1);
}
ch++;
}
}
close(fd);
}
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Weird behavior in receive_data function
On Thu, 2009-06-11 at 07:11 +0800, Daniel.Li wrote:
Dear List,
I'm trying to get diff/removed data and it's offset out. So I write a
functions in receive_data. When I run backup, I found there is a weird
behavior which I don't understand.
i = recv_token(f_in, data) will receive (i = -1, offset2 = 0) some
where in the middle of the transfer procedure. That's to say, it's going
to transfer the first data block from sender,
Sorry, It's to say copy the first block from receiver, NOT first data
block from sender.
but I have been
transferring almost half of the file. I noticed it was 92nd block
(sum.blength = 700) .
Hope I have a clear explanation. I'm really confused now. Any advice and
suggestion is well appreciated.
Below is my code segment to generate file for test.
My Procedure (file 1 and file 2 have the same name, just binary
difference):
Step 1: backup File 1
Step 2: backup Fil 2
--
Daniel Li
File 1:
This is the first time backup file.
count = 1;
nSize = 1;
nBlock = 256;
nInterval = 4;
nAdjust = 256;
printf(\nstart to build origin files!\n);
ch = 0;
for(l = 0; l count; l++)
{
memset(buffer,0, 256);
sprintf(buffer,media-test-3MB-file/origin/test/test%02d.dat,l);
printf(writting %s \n,buffer);
fd = open(buffer, O_RDWR|O_CREAT|O_EXCL,
S_IRUSR|S_IWUSR);
for(lll = 0; lll nSize; lll++)
{
for(mmm = 0; mmm nAdjust; mmm++)
{
for(nnn = 0; nnn nBlock; nnn++)
{
write_fhead_char(fd, ch, 1);
}
ch++;
}
}
}
printf(origin files done!\n);
File 2:
This is the second time backup. when transferring this file, I met the
trouble.
printf(\nstart to build added data files!\n);
ch = 0;
for(l = 0; l count; l++)
{
memset(buffer,0, 256);
sprintf(buffer,media-test-3MB-file/add/test/test%02d.dat,l);
printf(writting %s \n,buffer);
fd = open(buffer, O_RDWR|O_CREAT|O_EXCL,
S_IRUSR|S_IWUSR);
for(lll = 0; lll nSize + 1; lll++)
{
for(mmm = 0; mmm nAdjust; mmm++)
{
if(mmm % nInterval == 0)
{
for(nnn = 0; nnn nBlock;
nnn++)
{
write_fhead_char(fd,
ch, 1);
}
}
for(nnn = 0; nnn nBlock; nnn++)
{
write_fhead_char(fd, ch, 1);
}
ch++;
}
}
close(fd);
}
--
Daniel Li
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Key for high CPU usage
Dear List, I'm trying to take a closer look at rsync code, and found when we run daemon, it will take a lot of CPU (400Mhz). So I'm interested in Which part of rsync code on ver 3.0.5 consuming CPU a lot? Can anyone here help to lighten me up? So I can try to improve the performance or low the CPU usage. I suspect that there are a few factors, which might related with CPU usage: rolling checksum/Disk IO(a slide window has been implemented), read or write? Hope I can find some info here! Thanks in advance! -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --daemon. Can I open more than one instances?
On Mon, 2009-06-08 at 22:39 -0500, Rahul Nabar wrote: If you're not using different partitions on different disks, then doing anything in parallel is probably going to slow it down (reading/writing all over the disk, means more seektime, which means waiting). No, its just a single partition. But the single disk is a hardware RAID5 at one end. (The other backup end is a SATA drive) Is it still pretty much typical that the disk is the bottleneck? Not CPU/memory etc.? In spite of the diff computation etc. does disk seek remain the bottleneck usually. a) Try iometer/hdparm or other tools to get HD performance data; b) Moniter CPU and mem usage during backup; c) Moniter network usage; d) ... bla bla... I think you can test the system to find out what is the bottleneck. I run a daemon on my Nasbox, and the limit is cpu usage, cause it's 400Mhz. I this case I suggest to use samba, just mount to your Laptop and backup to mount point. Well, just my opinion, network and disk is probably the bottleneck for the system like yours. If you have got those data, then you will find the way to go. If it's disk, as Leon said, backup to different disks, or why NOT try raid 0/raid 10, or something else? -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
what kind attrs are going to be transfered by rsync?
Dear list, I have met this error. I wanna know what attrs are transfer by rsync program? sent 5045 bytes received 262853 bytes 107159.20 bytes/sec total size is 130560 speedup is 0.49 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1535) [generator=3.0.5] Any help is appreciated. Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: what kind attrs are going to be transfered by rsync?
It seems there is a discussion long time ago. But little info. http://fixunix.com/tools/540757-do-not-reply-bug-3245-improve-rsyncs-error-messages-make-themclearer.html I noticed that it's status is assigned, Has it been improved? https://bugzilla.samba.org/show_bug.cgi?id=3245 On Thu, 2009-06-04 at 14:44 +0800, Daniel.Li wrote: Dear list, I have met this error. I wanna know what attrs are transfer by rsync program? sent 5045 bytes received 262853 bytes 107159.20 bytes/sec total size is 130560 speedup is 0.49 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1535) [generator=3.0.5] Any help is appreciated. Thanks. -- Daniel Li -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --daemon. Can I open more than one instances?
On Wed, 2009-06-03 at 20:57 -0500, Rahul Nabar wrote: Is there a way to speed up rsync by opening more than one daemon in parallel. daemon will create a process when a connection is established. So the limit is hardware. I use rsync --daemon to start rsync. I was wondering if opening more than one instance is recommended or feasible? I think multi-client can improve performance, but limit is the same as above. I know that for services like nfs for example I have had opening many instances improve performance. What kind of performance you are concerning, network/cpu/ram/disk IO ? I have 4 cores available It seems good :) so if there are any other parallization modes I'd be glad to know since I need to backup ~800GB data every day. Rsync can backup only differs. I think you can use this feature. -- Rahul -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Which function in generator trigger the file transfer?
Dear List, I checked the function generate_files, which might be simply stand for generator, but didn't find where triggers the file transfer. As receiver will use read_ndx_and_attrs to read iflags to judge if it's need to be transfered? If I'm wrong, please correct me. Can anyone help to point it out? Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which function in generator trigger the file transfer?
I'm new to rsync, just read little of codes right now. I have read the pipe line would be generator → sender → receiver. But I didn't find where rsync makes sure that the file is going to be transfered (time stamp/file checksum). A. Is there anyone familiar with the code, can give me some tips on a) where generator send the time stamp/file checksum b) where sender compared time stamp/file stamp c) where receiver make sure that it's a go mission? I think some function names is enough for me shorten the time and get a basic understanding of the code. Thanks in advance. On Tue, 2009-06-02 at 15:31 +0800, Daniel.Li wrote: Dear List, I checked the function generate_files, which might be simply stand for generator, but didn't find where triggers the file transfer. As receiver will use read_ndx_and_attrs to read iflags to judge if it's need to be transfered? If I'm wrong, please correct me. Can anyone help to point it out? Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to emulate rdiff behaviour
Dear Wayne, Excellent, that's really what I have expected! Is it stable now? Cause I have found that this feature seems to be unstable before ver 3.0.6. - Fixed a --read-batch hang when rsync is reading a batch file that was created from an incremental-recursion transfer. http://samba.anu.edu.au/ftp/rsync/src/rsync-3.0.6-NEWS On Fri, 2009-05-29 at 23:48 -0700, Wayne Davison wrote: On Thu, May 28, 2009 at 11:32:16AM -0400, Matt McCutchen wrote: - Each day, generate a reverse batch (by running rsync in the reverse direction with --only-write-batch) and then update the destination file. In order to avoid any inconsistency between the files when the reverse run is done and when the backup copy is done, you could use a second full backup copy (though someone using something like LVM could use snapshots instead). To be explicit, a two-dir method would start by duplicating the backup dir (let's assume to a separate host, though it would also work without bhost: in all these commands): rsync -av /backup/ bhost:/backup2/ Then a backup operation would be: rsync -av srchost:/src/ /backup/ rsync -av --only-write-batch=/batches/$DATE bhost:/backup2/ /backup/ rsync -av /backup/ bhost:/backup2/ And one more thing here: If you are going to prepare this batch file, it seems there will be double the workload of network, see below statements? Is that right? rsync -av --only-write-batch=/batches/$DATE bhost:/backup2/ /backup/ rsync -av /backup/ bhost:/backup2/ That results in the daily double-backup dirs and a batch that lets you get back to the prior day's files. A restore operation would require running a sequence of batch files into the /backup/ dir (without any backups running): rsync -av --read-batch=/batches/$DATE1 /backup/ rsync -av --read-batch=/batches/$DATE2 /backup/ ... Then, after restoring the file(s), revert /backup to the most recent backup data: rsync -av bhost:/backup2/ /backup/ ..wayne.. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to emulate rdiff behaviour
On Sun, 2009-05-31 at 14:34 -0400, Matt McCutchen wrote: And one more thing here: If you are going to prepare this batch file, it seems there will be double the workload of network, see below statements? Is that right? rsync -av --only-write-batch=/batches/$DATE bhost:/backup2/ /backup/ rsync -av /backup/ bhost:/backup2/ Not if you put /backup and /backup2 on the same machine by dropping the bhost: from those commands, as Wayne mentioned. He included it just to point out the possibility of having those dirs on different machines. Humm... I have a NAS storage device, which is bhost, and I found that rsync will NOT simple transfer exactly the diff size of the contents, see below my explanation. I have done a simple test: 1) backup a file; 2) remove 256 bytes in this file; 3) backup again, this algorithm will transfer 1144 bytes of diff data, which is about 4 times the size of original removed data block. We are very much concerned on workload of network. As we have only 256 diff bytes, If we use batch files with bhost, then we might be need to transfer about 8 times the size of original removed data block. Well, I think I don't have much detailed knowledge on rsync diff algorithm. So I raised this question here. Maybe the transfer size will be different depending on the offset and size of removed data block. Can anyone here help to explain it? Or is there any info that I have missed or any report/document/evaluation? Please help to point me to these URLs. Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to emulate rdiff behaviour
On Sun, 2009-05-31 at 23:11 -0400, Matt McCutchen wrote: On Mon, 2009-06-01 at 07:10 +0800, Daniel.Li wrote: On Sun, 2009-05-31 at 14:34 -0400, Matt McCutchen wrote: And one more thing here: If you are going to prepare this batch file, it seems there will be double the workload of network, see below statements? Is that right? rsync -av --only-write-batch=/batches/$DATE bhost:/backup2/ /backup/ rsync -av /backup/ bhost:/backup2/ Not if you put /backup and /backup2 on the same machine by dropping the bhost: from those commands, as Wayne mentioned. He included it just to point out the possibility of having those dirs on different machines. Humm... I have a NAS storage device, which is bhost, Does it have to be that way? If you run Wayne's entire command sequence on the NAS, you won't have a bhost and you'll avoid doubling the network usage. But the source folder is on my laptop, about 30GB size, and I need backup to NAS. So if do this batch file on NAS without network usage, we have to copy a previous backup internally on NAS. I have to say 30GB internal copy really takes a lot of time. What do u think? -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: how to output verbose information to console when run rsync --daemon?
I used --log-file option, but I just found those connectbla bla... info. No other valuable info like rsync client with -v options. Can anyone help here ? On Wed, 2009-05-06 at 11:42 +0800, Daniel.Li wrote: Dear List, option -v can output information to console. But it seems no effect with rsync --daemon. So, how can I output verbose information to console when running with command rsync --daemon I also tried log file. Seems didn't work. Or is there any other way to read those information, Thanks in advance. -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: weird access function in rsync code.
On Fri, 2009-05-22 at 22:06 +0800, Daniel.Li wrote: On Fri, 2009-05-22 at 13:10 +0200, Paul Slootman wrote: On Fri 22 May 2009, Daniel.Li wrote: On Fri, 2009-05-22 at 10:47 +0200, Paul Slootman wrote: access(/home/admin/test,F_OK) Result: Failed with -1, just mean the test folder is NOT existed, but it does exist. Perhaps at that point rsync has chroot()ed somewhere else? I checked uid/gid just before my code, uid=0 gid=0. And I use hard coded path. None of those are relevant to my question... chroot() != setuid() Humm... Got it. http://en.wikipedia.org/wiki/Chroot That's great. Seems to be the root cause. Report back when I have news. Late today :) Sorry for the late reply. Had a fever, recently :(, but ok now :) I have found it in clientserver.c. Thanks Paul :) Thanks you :) Paul -- Daniel Li -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Does rsync detect file corruption? -- hard link
On Sun, 2009-05-24 at 16:04 +0200, Mac User FR wrote: Hard-linking an unchanged dir takes very few place. What if a video editor? Lots of work with video files, which is very large, about 500MB per file. Editor only delete or rearrange frames in that file. And then it will be back up 500MB again. In this case rsync can handle properly. I think. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Does rsync detect file corruption? -- hard link
On Sun, 2009-05-24 at 22:13 -0400, Matt McCutchen wrote: On Mon, 2009-05-25 at 10:09 +0800, Daniel.Li wrote: On Mon, 2009-05-25 at 09:58 +0800, Daniel.Li wrote: What if a video editor? Lots of work with video files, which is very large, about 500MB per file. Editor only delete or rearrange frames in that file. And then it will be back up 500MB again. In this case rsync can handle properly. I think. I mean in this case, rsync can't handle properly. It will backup 500MB again. It'll not save any disk space. If I'm wrong, please correct me. That's correct. When small changes are made to big files, rsync reduces network usage but not disk usage. rdiff-backup ( http://www.nongnu.org/rdiff-backup ) reduces disk usage by storing deltas in the destination, but then you need rdiff-backup to recover. But these are some limitations with rdiff-backup, well, just list what I concerned. a) does not support files greater than 2 GB' b) It seems quite slow (~0.5+MBps), but I didn't test it myself (http://www.nongnu.org/rdiff-backup/FAQ.html#speed ). -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
weird access function in rsync code.
Dear List, When I use access function to find out if the folder exist in rsync code. I found that it's will NOT act as normal function. I have tried 2 test: a) embedded in rsync function int recv_files(int f_in, char *local_name) and hard coded with following line. access(/home/admin/test,F_OK) Result: Failed with -1, just mean the test folder is NOT existed, but it does exist. b) simple hello world program OK. Result: return 0, folder exist. Right now, I can't figure out what's wrong. Can anyone give a tip? Thanks in advance. -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: weird access function in rsync code.
Really weird! root folder is OK, but others returned -1. Why? I'm in recv_files access / ret = 0 I'm in recv_files access /home ret = -1 I'm in recv_files access /home/admin ret = -1 I'm in recv_files access /home/admin/test ret = -1 Thoese folder does exist. I use rsync --daemon to launch the program and triggered with another rsync client. drwxr-xr-x 7 root root 4096 2009-03-31 16:29 home drwxr-xr-x 6 admin admin 4096 2009-05-22 15:24 admin drwxrwxrwx 3 admin admin 4096 2009-05-22 15:27 test BTW, rsync version 3.0.5. Any help is really appreciated. On Fri, 2009-05-22 at 16:14 +0800, Daniel.Li wrote: Dear List, When I use access function to find out if the folder exist in rsync code. I found that it's will NOT act as normal function. I have tried 2 test: a) embedded in rsync function int recv_files(int f_in, char *local_name) and hard coded with following line. access(/home/admin/test,F_OK) Result: Failed with -1, just mean the test folder is NOT existed, but it does exist. b) simple hello world program OK. Result: return 0, folder exist. Right now, I can't figure out what's wrong. Can anyone give a tip? Thanks in advance. -- Daniel -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: weird access function in rsync code.
On Fri, 2009-05-22 at 10:47 +0200, Paul Slootman wrote: On Fri 22 May 2009, Daniel.Li wrote: When I use access function to find out if the folder exist in rsync code. I found that it's will NOT act as normal function. I have tried 2 test: a) embedded in rsync function int recv_files(int f_in, char *local_name) and hard coded with following line. access(/home/admin/test,F_OK) Result: Failed with -1, just mean the test folder is NOT existed, but it does exist. Perhaps at that point rsync has chroot()ed somewhere else? I checked uid/gid just before my code, uid=0 gid=0. And I use hard coded path. I also check F_OK, it's 0. What could cause it fail? Really have no idea now. Still Google.. Hope someone give me some tips here. Paul -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: weird access function in rsync code.
On Fri, 2009-05-22 at 13:10 +0200, Paul Slootman wrote: On Fri 22 May 2009, Daniel.Li wrote: On Fri, 2009-05-22 at 10:47 +0200, Paul Slootman wrote: access(/home/admin/test,F_OK) Result: Failed with -1, just mean the test folder is NOT existed, but it does exist. Perhaps at that point rsync has chroot()ed somewhere else? I checked uid/gid just before my code, uid=0 gid=0. And I use hard coded path. None of those are relevant to my question... chroot() != setuid() Humm... Got it. http://en.wikipedia.org/wiki/Chroot That's great. Seems to be the root cause. Report back when I have news. Late today :) Thanks you :) Paul -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
2.6.9 and 3.0.5 sync error
Dear List, I'm using rsync to sync my data, see below command. And I met this error? I think it might be due to 2.6.9 and 3.0.5 compatible issue. So how to solve the problem? Thanks. rsync -vrtopg --progress --delete --password-file=rsync-password /path/on/pc r...@192.168.0.202::destination building file list ... 151804 files to consider rsync: push_dir#1 / (in maps) failed: Permission denied (13) rsync error: errors selecting input/output files, dirs (code 3) at main.c(486) [receiver=2.6.9] rsync: connection unexpectedly closed (4 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.5] -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: 2.6.9 and 3.0.5 sync error
On Wed, 2009-05-20 at 15:35 +0200, Mac User FR wrote: This seems for me an autorisation problem rsync: push_dir#1 / (in maps) failed: Permission denied (13) Verify that you have enough privileges to read the data you are trying to. I have tried root privilege on the both side :( Le 20 mai 09 à 13:01, Daniel.Li a écrit : Dear List, I'm using rsync to sync my data, see below command. And I met this error? I think it might be due to 2.6.9 and 3.0.5 compatible issue. So how to solve the problem? Thanks. rsync -vrtopg --progress --delete --password-file=rsync-password /path/on/pc r...@192.168.0.202::destination building file list ... 151804 files to consider rsync: push_dir#1 / (in maps) failed: Permission denied (13) rsync error: errors selecting input/output files, dirs (code 3) at main.c(486) [receiver=2.6.9] rsync: connection unexpectedly closed (4 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.5] -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- USI/SH RD Dept. Universal Scientific Industrial (Shanghai) Co., Ltd Fax: +86-21-58967931Tel: +86-21-58966996 ext. 1400 42th building, 1387 Zhang Dong road, Pudong New Area, Shanghai, P.R. China 201203 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Delta backup program planned (problem met for adding files)
Dear List, Currently, I'm trying to implement a new feature to rsync. Delta back: Well, it simply just backups diff blocks, and the program is able to restore to any backup points. Current problem met for adding files! But I met a newbie problem here. I'm going to add six files, which are diff.h/diff.c del.h/del.c and merge.h/merge.c. I have modified Makefile.in. When I add an function, compiler prints some error, see below: And I have found that proto.h is generated automatically. So what I have missed here in configuration (adding those six extra files) I'm really appreciated any help form you. Thanks in advance. gcc -std=gnu99 -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -c diff.c -o diff.o In file included from rsync.h:915, from diff.h:27, from diff.c:25: proto.h:62: warning: ‘struct diff_file_list’ declared inside parameter list proto.h:62: warning: its scope is only this definition or declaration, which is probably not what you want diff.c: In function ‘init_diff_idx’: diff.c:56: warning: unused parameter ‘pdiffidx_file’ diff.c: At top level: diff.c:65: error: conflicting types for ‘update_diff_idx’ proto.h:62: error: previous declaration of ‘update_diff_idx’ was here diff.c: In function ‘update_diff_idx’: diff.c:65: warning: unused parameter ‘pdifflist’ diff.c: In function ‘check_diff_idx’: diff.c:74: warning: unused parameter ‘pdiffidx_file’ make: *** [diff.o] Error 1 -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Delta backup program planned (problem met for adding files)
On Mon, 2009-05-11 at 10:05 +0800, Daniel.Li wrote: When I add an function, compiler prints some error, see below: And I have found that proto.h is generated automatically. So what I have missed here in configuration (adding those six extra files) OK, I have found the problem. If I move the defination of struct diff_file_list to rsync.h, then it's OK. I think it might be something to do with OLDNEWS #line 642 since 3.0.0, see below: - The proto.h file is now built using a simple perl script rather than a complex awk script, which proved to be more widely compatible. but I don't know why, can anyone help to explain? Thanks. I'm really appreciated any help form you. Thanks in advance. gcc -std=gnu99 -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -c diff.c -o diff.o In file included from rsync.h:915, from diff.h:27, from diff.c:25: proto.h:62: warning: ‘struct diff_file_list’ declared inside parameter list proto.h:62: warning: its scope is only this definition or declaration, which is probably not what you want diff.c: In function ‘init_diff_idx’: diff.c:56: warning: unused parameter ‘pdiffidx_file’ diff.c: At top level: diff.c:65: error: conflicting types for ‘update_diff_idx’ proto.h:62: error: previous declaration of ‘update_diff_idx’ was here diff.c: In function ‘update_diff_idx’: diff.c:65: warning: unused parameter ‘pdifflist’ diff.c: In function ‘check_diff_idx’: diff.c:74: warning: unused parameter ‘pdiffidx_file’ make: *** [diff.o] Error 1 -- Daniel -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Prepared meta directory for one to many sync
On Wed, 2009-05-06 at 09:20 +0200, Lukas Macura wrote: Hi to all, sorry, but I am not subscribed to this list. I have a question, because I did not find any answer for my problem. If you have so, please let me know! We use rsync to synchronise one-to-many workstations. We use our obnovang software (http://obnova.bp.opf.slu.cz/). You can read more info on this page, but in general, we are doing backups and refresh of many workstations (aprox. 200) which are in our PC labs. So we prepare one workstation, rsync data from local filesystem to rsync server and next, all stations will connect to rsync server and refresh their local filesystems to be copy of server. Everything works great, thank you for great rsync code :) but we have some suggestions how to improve speed of our process. - We cannot enable compression because file is compressed just when it is sended. If 100PCs are connected to rsync server and we will turn on compression, server will die. - We would want to use compression because some links are slow and compression would increase speed. - Same to partial file transfer. We use --whole-file , because if server would compute hashes each time when client connect, server will die.. Why not try to use multi-server. for example, this is a central server. central server --- backup to three mirror server,named with #1, #2, #3 #1 mirror server --- 50 PC #2 mirror server --- 50 PC #3 mirror server --- 50 PC #4 mirror server --- 50 PC It might reduce the work load of central server. What do u think? I know our scenario is similar to some rsync download mirror and maybe it is solved somewhere. But I did not find any solution yet. Maybe it would be great if rsync have some meta directory, where it would prepare all infos about stored files. Like hashes, precompressed files, etc. This meta directory could be created or refreshed by some options. After creating and pointing to that directory, entire mirroring process could be much more faster. Maybe: $ rsync -r --prepare-meta /tmp/meta ./ $ rsync -r --meta-dir /tmp/meta/ ./ rsync://somewhere/ Please is it theoreticaly possible ? Or is there some problem to precompress and prehash files? Thank you for any answer, Lukas Macura -- USI/SH RD Dept. Universal Scientific Industrial (Shanghai) Co., Ltd Fax: +86-21-58967931Tel: +86-21-58966996 ext. 1400 42th building, 1387 Zhang Dong road, Pudong New Area, Shanghai, P.R. China 201203 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
how to output verbose information to console when run rsync --daemon?
Dear List, option -v can output information to console. But it seems no effect with rsync --daemon. So, how can I output verbose information to console when running with command rsync --daemon I also tried log file. Seems didn't work. Or is there any other way to read those information, Thanks in advance. -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Code study]should we remove if (motd *motd) section?
Dear List,
I'm confused when we will run into if (motd *motd)?
As I have found that Globals is set 0 during initialization, and I
didn't find anywhere else assign the value.
If so, should we remove this section?
Or there might be some where I missed? Please correct me, if I'm wrong.
Thanks in advance.
clientserver.c #line 147~160
if (!am_client) {
motd = lp_motd_file();
if (motd *motd) {
FILE *f = fopen(motd,r);
while (f !feof(f)) {
int len = fread(buf, 1, bufsiz - 1, f);
if (len 0)
write_buf(f_out, buf, len);
}
if (f)
fclose(f);
write_sbuf(f_out, \n);
}
}
static void init_globals(void)
{
memset(Globals, 0, sizeof Globals);
}
--
Daniel
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Code study]should we remove if (motd *motd) section?
On Mon, 2009-04-27 at 12:41 +0200, Paul Slootman wrote:
On Mon 27 Apr 2009, Daniel.Li wrote:
I'm confused when we will run into if (motd *motd)?
As I have found that Globals is set 0 during initialization, and I
didn't find anywhere else assign the value.
clientserver.c #line 147~160
if (!am_client) {
motd = lp_motd_file();
^^
if (motd *motd) {
It's being assigned the line above the (motd *motd) part!
Sorry, I forgot to paste below macros. Yes, it's assigned, but this
function just returns default() or global values. But it seems never
assigned.
loadparam.c #line 394
FN_GLOBAL_STRING(lp_motd_file, Globals.motd_file)
loadparam.c #line 374~375
#define FN_GLOBAL_STRING(fn_name,ptr) \
char *fn_name(void) {return(*(char **)(ptr) ? *(char **)(ptr) : );}
So the function will return or it's assigned value.
char *lp_motd_file(void)
{
return(*(char **)(Globals.motd_file) ? *(char **)(Globals.motd_file) :
);
}
Paul
--
Daniel
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Code study]should we remove if (motd *motd) section?
On Mon, 2009-04-27 at 19:21 +0800, Daniel.Li wrote:
On Mon, 2009-04-27 at 12:41 +0200, Paul Slootman wrote:
On Mon 27 Apr 2009, Daniel.Li wrote:
I'm confused when we will run into if (motd *motd)?
As I have found that Globals is set 0 during initialization, and I
didn't find anywhere else assign the value.
clientserver.c #line 147~160
if (!am_client) {
motd = lp_motd_file();
^^
if (motd *motd) {
It's being assigned the line above the (motd *motd) part!
Sorry, I forgot to paste below macros. Yes, it's assigned, but this
function just returns default() or global values. But it seems never
assigned.
What I mean is Globals.motd_file is NEVER assigned. When we use
default . Then we still will NOT go into if (motd *motd), as
*motd == 0.
What do u think?
loadparam.c #line 394
FN_GLOBAL_STRING(lp_motd_file, Globals.motd_file)
loadparam.c #line 374~375
#define FN_GLOBAL_STRING(fn_name,ptr) \
char *fn_name(void) {return(*(char **)(ptr) ? *(char **)(ptr) : );}
So the function will return or it's assigned value.
char *lp_motd_file(void)
{
return(*(char **)(Globals.motd_file) ? *(char **)(Globals.motd_file) :
);
}
Paul
--
Daniel
Daniel
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Code study]should we remove if (motd *motd) section?
On Mon, 2009-04-27 at 13:34 +0200, Petr Uzel wrote:
On Mon, Apr 27, 2009 at 07:27:38PM +0800, Daniel.Li wrote:
On Mon, 2009-04-27 at 19:21 +0800, Daniel.Li wrote:
On Mon, 2009-04-27 at 12:41 +0200, Paul Slootman wrote:
On Mon 27 Apr 2009, Daniel.Li wrote:
I'm confused when we will run into if (motd *motd)?
As I have found that Globals is set 0 during initialization, and I
didn't find anywhere else assign the value.
clientserver.c #line 147~160
if (!am_client) {
motd = lp_motd_file();
^^
if (motd *motd) {
It's being assigned the line above the (motd *motd) part!
Sorry, I forgot to paste below macros. Yes, it's assigned, but this
function just returns default() or global values. But it seems never
assigned.
What I mean is Globals.motd_file is NEVER assigned. When we use
default . Then we still will NOT go into if (motd *motd), as
*motd == 0.
I'd guess it is assigned in the code that parses configuration file
(rsyncd.conf).
What if you set 'motd file = /etc/rsync.motd' in /etc/rsyncd.conf?
Humm... Thanks for the point.
load_config - lp_load - pm_process - do_parameter
do_parameter compares motd file, like a key, then assigned
Globals.motd_file = /etc/rsync.motd
Well, NOW, take a close look at motd file, I know, it's something like
welcome message.
And the rsync protocol 30 only support 1024 chars of welcome message.
Thanks :)
--
Best regards / s pozdravem
Petr Uzel, Packages maintainer
-
SUSE LINUX, s.r.o. e-mail: pu...@suse.cz
Lihovarská 1060/12 tel: +420 284 028 964
190 00 Prague 9 fax: +420 284 028 951
Czech Republic http://www.suse.cz
--
Daniel
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Code study]append_mode definitions
On Mon, 2009-04-27 at 06:44 -0700, Wayne Davison wrote: On Mon, Apr 27, 2009 at 10:33:02AM +0800, Daniel.Li wrote: Can anyone help to explain definitions of variable append_mode? There are two different type of appending (see the man page) depending on how new rsync is and how much of the file should be checksummed. We are studying ver 3.0.5. So there are three states: 1) append_mode = 0, NONE append mode. 2) append_mode = 1, --append If file size on the receiver is the same or longer than the size on the sender, the file is skipped 3) append_mode = 2, --append-verify This works just like the --append option, but the existing data on the receiving side is included in the full-file checksum verification step The option is also disabled if a file is being fixed during a redo event. I didn't get it. Can u help to explain a little bit more? Thanks. But I still doesn't know why append_mode = -append_mode in those files? generate.c #line2102,2127 append_mode = -append_mode; receive.c #line503,513 append_mode = -append_mode; send.c #line248,254 append_mode = -append_mode; ..wayne.. --- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Code study]append_mode definitions
Dear List,
Can anyone help to explain definitions of variable append_mode?
I have found this variable is changed in many places, and quite
confused.
option.c #line549~550
{append-verify,0, POPT_ARG_VAL,append_mode, 2, 0, 0 },
{no-append,0, POPT_ARG_VAL,append_mode, 0, 0, 0 },
#line1183~1188
case OPT_APPEND:
if (am_server)
append_mode++;
else
append_mode = 1;
break;
batch.c #line153~156
if (tweaked_append)
append_mode = 1;
else if (tweaked_append_verify)
append_mode = 2;
compat.c #line191~192
if (append_mode == 1)
append_mode = 2;
generate.c #line2102,2127
append_mode = -append_mode;
receive.c #line503,513
append_mode = -append_mode;
send.c #line248,254
append_mode = -append_mode;
--
Daniel
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Any program flow or software spec for Rsync
On Fri, 2009-04-24 at 07:36 -0700, Wayne Davison wrote: On Tue, Apr 21, 2009 at 02:47:41PM +0800, Daniel.Li wrote: I'm trying to take a look at rsync from code-level. Is there any software spec or program flow that I can get, which might give me a basic understanding about the code? The only thing I can think of is an old how-rsync-works page: http://rsync.samba.org/how-rsync-works.html Beyond that, rsync's protocol and/or program flow is not documented anywhere. I have been looking at the code for almost about 3 days. :( It seems NOT that easy to understand, especially protocol 30, well something binary, global variable,...bla...bla But I'm still wanna take a closer look at the code. So what I'm planning is to just document what I have seen and think. I'll prepare a wiki page and update when I have a further understanding of the code. It's almost 23:30 :), I'll create the page and provide the link tomorrow. I hope that I can get more help on this. Any advice/suggestion/help is mostly appreciated. Thanks. ..wayne.. -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: looking for superlifter souce code and related information
On Fri, 2009-04-24 at 07:34 -0700, Wayne Davison wrote: On Tue, Apr 21, 2009 at 10:39:58AM +0800, Daniel.Li wrote: I just googled superlifter and found below link, but I can't get any source code. Sadly, superlifter never achieved lift-off, so there was never any source code for it. I'm sorry to hear that. ..wayne.. -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Any program flow or software spec for Rsync
Is there any protocol standard or specs that I can get? Draft is also fine. Thanks. On Tue, 2009-04-21 at 14:47 +0800, Daniel.Li wrote: Dear All, I'm trying to take a look at rsync from code-level. Is there any software spec or program flow that I can get, which might give me a basic understanding about the code? Thanks in advance! -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Code study] lp_motd_file defination? Help
Dear List,
Currently, I read this in clientserver.c, line 148.
motd = lp_motd_file();
I have googled, but didn't the definition of char *lp_motd_file(void).
Can anyone help to explain the following code segment. Many thanks.
#line 147 ~ 160, in clientserver.c, version 3.0.5
if (!am_client) {
motd = lp_motd_file();
if (motd *motd) {
FILE *f = fopen(motd,r);
while (f !feof(f)) {
int len = fread(buf, 1, bufsiz - 1, f);
if (len 0)
write_buf(f_out, buf, len);
}
if (f)
fclose(f);
write_sbuf(f_out, \n);
}
}
$ grep lp_motd_file . -R
./proto.h:char *lp_motd_file(void);
./loadparm.c:FN_GLOBAL_STRING(lp_motd_file, Globals.motd_file)
./clientserver.c: motd = lp_motd_file();
--
Daniel
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Code study] lp_motd_file defination? Help
On Wed, 2009-04-22 at 15:54 +0800, Daniel.Li wrote:
Dear List,
Currently, I read this in clientserver.c, line 148.
motd = lp_motd_file();
I have googled, but didn't the definition of char *lp_motd_file(void).
Can anyone help to explain the following code segment. Many thanks.
I got this, just take a closer look at those macros
FN_GLOBAL_STRING(lp_motd_file, Globals.motd_file)
#define FN_LOCAL_STRING(fn_name,val) \
char *fn_name(int i)
{return((LP_SNUM_OK(i)pSERVICE(i)-val)?pSERVICE(i)-val :
(sDefault.val?sDefault.val:));}
So, char *lp_motd_file(void) is as below. Well, I still didn't have a
clear picture about below function. Any help is mostly appreciated.:)
char *lp_motd_file(int i) {
return((LP_SNUM_OK(i)pSERVICE(i)-Globals.motd_file)?
pSERVICE(i)-Globals.motd_file :
(sDefault.Globals.motd_file?sDefault.Globals.motd_file:));
}
--
Daniel
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Code study] lp_motd_file defination? Help
On Wed, 2009-04-22 at 10:14 +0200, Paul Slootman wrote: On Wed 22 Apr 2009, Daniel.Li wrote: Currently, I read this in clientserver.c, line 148. motd = lp_motd_file(); I have googled, but didn't the definition of char *lp_motd_file(void). $ grep lp_motd_file . -R ./proto.h:char *lp_motd_file(void); ./loadparm.c:FN_GLOBAL_STRING(lp_motd_file, Globals.motd_file) So look in loadparm.c ... I'm guessing this is for some school course? If you're reasonably able to read code, that code snippet shouldn't be hard to understand, and if not, ask your teacher, not us :-) Sorry, I'm in a hurry. I just wanna get a full picture of program flow and basic APIs of algorithm. We are planning to implement a feature, which will enable us to restore to any backup point. Is it possible for us to do so?\ Any idea is mostly appreciated. Thanks. Paul -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: looking for superlifter souce code and related information
There is a thread palnning for superlifter http://lists.samba.org/archive/rsync/2002-August/003361.html But I don't know if there is any action later? Anyone knows? Or maybe I have take a look at rZync :) On Tue, 2009-04-21 at 10:39 +0800, Daniel.Li wrote: Dear List, I just googled superlifter and found below link, but I can't get any source code. http://superlifter.sourceforge.net/ Any help or info on this is mostly appreciated. And sorry for the trouble, if it's NOT allowed to post here. THanks. -- Daniel -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Any program flow or software spec for Rsync
Dear All, I'm trying to take a look at rsync from code-level. Is there any software spec or program flow that I can get, which might give me a basic understanding about the code? Thanks in advance! -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
looking for superlifter souce code and related information
Dear List, I just googled superlifter and found below link, but I can't get any source code. http://superlifter.sourceforge.net/ Any help or info on this is mostly appreciated. And sorry for the trouble, if it's NOT allowed to post here. THanks. -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How many differs in rsync algorithm from Andrew Tridgell's thesis
Is there anyone can help me on this? On Thu, 2009-04-16 at 10:13 +0800, Daniel.Li wrote: Dear List, I would like to take a look at rsync's algorithm. As newbie on this algorithm, I just downloaded Andrew Tridgell's original thesis in 1999. So my question is How many differs in rsync algorithm from Andrew Tridgell's thesis. Thanks in advance. Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How many differs in rsync algorithm from Andrew Tridgell's thesis
Dear List, I would like to take a look at rsync's algorithm. As newbie on this algorithm, I just downloaded Andrew Tridgell's original thesis in 1999. So my question is How many differs in rsync algorithm from Andrew Tridgell's thesis. Thanks in advance. -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--disable-iconv default configuration
Dear List, I have a quick question about rsync configuration. If I just simply use ./configure, does --iconv supported by server? --disable-iconv disable rsync's --iconv option Thanks. -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --disable-iconv default configuration
On Mon, 2009-03-16 at 14:08 +0800, Daniel.Li wrote: Dear List, I have a quick question about rsync configuration. If I just simply use ./configure, does --iconv supported by server? --disable-iconv disable rsync's --iconv option OK, the above problem has been solved. --iconv is enable with default configuration. But I have found this error Invalid or incomplete multibyte or wide character while transferring my files. Is it possible to convert these character encoding into utf8 on the fly with rsync? How can I fix the problem? Any hint is well appreciated, thanks. Thanks. -- Daniel -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --disable-iconv default configuration
On Mon, 2009-03-16 at 14:42 +0800, Daniel.Li wrote: On Mon, 2009-03-16 at 14:08 +0800, Daniel.Li wrote: Dear List, I have a quick question about rsync configuration. If I just simply use ./configure, does --iconv supported by server? --disable-iconv disable rsync's --iconv option OK, the above problem has been solved. --iconv is enable with default configuration. But I have found this error Invalid or incomplete multibyte or wide character while transferring my files. Is it possible to convert these character encoding into utf8 on the fly with rsync? How can I fix the problem? Any hint is well appreciated, thanks. --iconv=gb2312 works fine for me. Excellent software! :) Thanks. -- Daniel -- Daniel -- USI/SH RD Dept. Universal Scientific Industrial (Shanghai) Co., Ltd Fax: +86-21-58967931Tel: +86-21-58966996 ext. 1400 42th building, 1387 Zhang Dong road, Pudong New Area, Shanghai, P.R. China 201203 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync Over SSH (Windows XP)
On Sat, 2009-03-14 at 23:41 +1100, David Overton wrote: Hi Henri, 2009/3/14 henri he...@stmargarets.school.nz: In essence I would like to determine if rsync a good choice when it comes to the backup of files on a Windows XP machine? Suggestions such as the following will all be warmly welcomed : (1) Do not run rsync over ssh, keep it simple. (2) Do not use rsync on Windows XP. (3) Try xyz commercial solution/service instead, it works great! (4) Use xyz open source solution it is fantastic! (4) It works but it is difficult, prepare your self for punishment. (5) No problem, just follow these instructions you will be running in 1 day. (6) It works, just make sure you install the xyz patch. (7) Not enough information, it depends on various other factors. ...etc I am aware that there is the popular Cygwin project which provides a super simple way of installing both rsync and OpenSSH on Windows systems. There's no native port of rsync on Windows so using the Cygwin port is the way to go. If you don't want a full Cygwin installation there is an excellent package called cwRsync which contains just what you need at http://www.itefix.no/i2/node/10650. The same site provides an OpenSSH package called CopSSH which is designed to work well with cwRsync. Both packages come with Windows-style installers that set up the SSH/rsync daemons as Windows services. If you want a commercial backup solution, you might be interested in BackupAssist (www.backupassist.com) which now supports backup via rsync as well as other methods. (Disclaimer: I work for the company that develops BackupAssist.) BackupAssist provides various pre-defined schemes for regular scheduled backups with history. No need to fiddle around with scripts and rsync options, just enter your server details, select a scheme then select which files/directories you want to backup. There's a 30 day free trial period if you want to try it out. Other features: - Supports rsync daemon, rsync over ssh, and rsync daemon via ssh tunnel - Backup history uses hard links for files that haven't changed between backups, similar to rsnapshot. (Our marketing department calls this single instance store.) - Support for Volume Shadow Copy service (VSS) for Windows 2003 and later (not XP unfortunately). Humm... Excellent software. But I don't have enough money to buy :) I hope this doesn't sound too much like an ad, but you did ask about commercial solutions. Cheers, David -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to speed up rsync when haveing lots of files
On Tue, 2009-03-03 at 23:46 -0700, lewis butler wrote: On 3-Mar-2009, at 22:55, Daniel.Li wrote: -z is apparently affect the performance when CPU has a lower frequency, like 200MHz or so. When doing rsync, 100% cpu occupied, which limits network performance. You should not use -z at ALL with large numbers of small files. The increased latency of the compression/decompression will far exceed any time saved in transmission. You're not on a 300 baud modem, I assume? -z should be used when you are sending large files that are compressible in the first place (so not video or mp3s, for example), but it it only going to hurt you on small files. How small? Depends on your connection. My rule of thumb, based on nothing at all, is that the speed in megabits of my connection is the size in megabytes that I start worrying about compression. Compressible data over 15MB in size? OK, I will start to think about using -z. If I was on a T1? 1.5MB. ADSL? 700K. Arbitrary, perhaps, but it seems to have served me well. In fact, I found removing -z sped up transfers quite a bit on my LAN. Humm,,, u mentioned about average file size and network connection, which should taken into consideration of using -z option. Well, I think when developing this option, it must have some report or concerns. And after implement this feature, there will be also some kind of test report, or some stuff like that to say hello, this works. But I don't know where I can get those information:( BTW: I don't know more about --compress-level=NUM, how many levels we could use to set? See above. As for the number range, I've always assume a 0-9 range much like gzip, though that is not explicitly given in the man page. Yeah, if we had above information, then it's much closer to the answer. Even if we don't have the data, then we have the direction to test it out. BTW, currently, I have 100Mb local network. I need to get the best result of rsync. Currently, my data shows that total_file_size/time(include transfer and preparition), vary from 2.72~7.7MB/s So I wanna know how to maximize the performance both of large file (1GB patter file) and small file (MP3, about 2~5MB per file) Maybe MP3 shouldn't be considered small files, right? Do u guys have any suggestions? Any hint or advice is mostly appreciated. Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: How to speed up rsync when haveing lots of files
On Wed, 2009-03-04 at 13:41 +0100, Boniforti Flavio wrote: This question has been answered. Upgrade to 3.0.5 on both sides. Why has upgrading to 3.0.5 to be done *on both sides*? Any explanation?! As I'm running Debian Lenny (which has 3.0.3), do I have to switch to Debian Sid to get it upgraded? Well, I have tested rsync, which is based on 3.0.4 (client) and 3.0.5(Server). Is it necessary to upgrade to 3.0.5(Client)? I have just checked change history http://samba.anu.edu.au/ftp/rsync/src/rsync-3.0.5-NEWS , and have NOT found any main fix on performance issue. I'm also interested in Flavio's question about Why has upgrading to 3.0.5 to be done *on both sides*? Thanks, Flavio Boniforti PIRAMIDE INFORMATICA SAGL Via Ballerini 21 6600 Locarno Switzerland Phone: +41 91 751 68 81 Fax: +41 91 751 69 14 Url: http://www.piramide.ch E-mail: fla...@piramide.ch -- USI/SH RD Dept. Universal Scientific Industrial (Shanghai) Co., Ltd Fax: +86-21-58967931Tel: +86-21-58966996 ext. 1400 42th building, 1387 Zhang Dong road, Pudong New Area, Shanghai, P.R. China 201203 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to speed up rsync when haveing lots of files
On Tue, 2009-03-03 at 08:58 -0500, Charles Marcus wrote: On 3/3/2009 8:52 AM, m...@bortal.de wrote: unfortunatelly rsync is beeing REALLY slow and produces a high load when we try to sync lots of files (250 000 small files). What version of rsync are you using? # rsync --version rsync version 2.6.9 protocol version 29 Upgrade to 3.0.5 (on both ends) OK, besides this, is there any other way to improve the network performance? some thing like change the option or what? -- Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to speed up rsync when haveing lots of files
On Tue, 2009-03-03 at 21:26 -0700, lewis butler wrote: On 3-Mar-2009, at 20:33, Daniel.Li wrote: On Tue, 2009-03-03 at 08:58 -0500, Charles Marcus wrote: On 3/3/2009 8:52 AM, m...@bortal.de wrote: unfortunatelly rsync is beeing REALLY slow and produces a high load when we try to sync lots of files (250 000 small files). What version of rsync are you using? # rsync --version rsync version 2.6.9 protocol version 29 Upgrade to 3.0.5 (on both ends) OK, besides this, is there any other way to improve the network performance? some thing like change the option or what? Not so much. a large part of 3.0 is specifically to handle large numbers of files. I found there are quit lots of options might be related with the network performance, see bleow. Can anyone help explain in more detail. e.g. -z is apparently affect the performance when CPU has a lower frequency, like 200MHz or so. When doing rsync, 100% cpu occupied, which limits network performance. BTW: I don't know more about --compress-level=NUM, how many levels we could use to set? There might be others that limit/improve network performance. So an detailed explanation is mostly appreciated. Or there might have this kind of blog already, a link will also be OK. Thanks in advance. --- some options that might be related to network performance- -S, --sparsehandle sparse files efficiently -z, --compress compress file data during the transfer --compress-level=NUMexplicitly set compression level --skip-compress=LISTskip compressing files with a suffix in LIST --bwlimit=KBPS limit I/O bandwidth; KBytes per second --protocol=NUM force an older protocol version to be used -4, --ipv4 prefer IPv4 -6, --ipv6 prefer IPv6 -- Daniel Li 2009.03.04 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsycing very small files
1. On Fri, 2009-02-27 at 08:20 -0500, Mag Gam wrote: it works. But takes hours to do it. Was wondering if there was a faster way How much speed do u get to backup these files? Average? On Fri, Feb 27, 2009 at 8:04 AM, Paul Slootman paul+rs...@wurtel.net wrote: On Fri 27 Feb 2009, Mag Gam wrote: I have to rsync 200k files which range in size from 5kb to 800kb. Is there an optimal way to do this using rsync? or shall I use tar for this? Just do it, there's no reason why it shouldn't work. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- USI/SH RD Dept. Universal Scientific Industrial (Shanghai) Co., Ltd Fax: +86-21-58967931Tel: +86-21-58966996 ext. 1400 42th building, 1387 Zhang Dong road, Pudong New Area, Shanghai, P.R. China 201203 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Time slot back schema
Hi List, Does Rsync support time slot back schema? E.G. (same local folder and remote destination) 1) first backup at 2008-12-24 2) second backup at 2009-02-12 But something bad happens to 2209-02-12, I need to restore files to first backup. Is there any way to do so? Any help is mostly appreciated. Thanks. Daniel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
