Rsync for program loading on embedded platforms
Hi, Has anyone considered using the rsync algorithm for loading programs on embedded platforms? I.e. with low cpu power (say, 16-25 MHz ARM7) and low memory requirements (programs are 128 KB, and the rsync algorithm must use little ROM and RAM). Or is it just not worth the effort? I guess you'd want to at least modify the checksum calculations and use shorter and faster checksums (as the data to be transferred is smaller than on workstations), and you'd need that file overwrite patch that's been floating around. Regards, Greger -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
what does lp_log_file mean??
I have some problems when I read the code of the rsync. There are many reference such as lp_log_file, lp_uid, lp_include_from, etc. in the file clientserver.c and log.c . Who can tell what do they mean? Are they functions? Where I can find the particular information about them? Thanks! _ MSN Explorer: http://explorer.msn.com/lccn/ -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug reporting
On 1 Jun 2004, John Summerfield [EMAIL PROTECTED] wrote: The jitterbug link on http://rsync.samba.org/nobugs.html no longer works. I suggest it either be fixed or removed. Thanks, fixed. You make bug-reporting needlessly difficult, I think. I dislike the need to subscribe to a mailing-list and potentially receive lots of email that doesn't interest me. I have plenty of other email to keep me amused. I don't think you need to subscribe to post. I put the address directly on the nobugs page to make it easier for people to write to it. Did you have any other suggestions about how to make it better? The reason we took Jitterbug and faq-o-matic down is that people seemed to get help more promptly when they wrote to the list. What I wanted most to do is ensure you know about rsyncx and consider working with the authors to create a unified product that supports resource forks when built on OSX. See http://www.macosxlabs.org./rsyncx/rsyncx.html Their CVS repository is at http://www.opendarwin.org It seems a shame to have two projects where one will do. Well, sometimes there are reasons not to glom everything into one big program. -- Martin -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: I20 Drivers Crash system when used with Rsync
On 30 May 2004, Dennis R. Gesker [EMAIL PROTECTED] wrote: Note: I don't know if this is a problem withe I20 drivers or Rsync so I'm submitting to both the Kernel Bugzilla and the Rsync mailing list. I couldn't find a bugzilla for Rsync. I hope this was the correct way to submit this issue. Distribution: Debian Hardware Environment: Intel 850MV Mother board, Pentium 4 processor, 1Gig of RAM, Adaptec 2400A RAID Controler. Both the motherboard and Controller card have the most recent BIOS/firmware installed. The Adaptec card is capable of RAID configuration but currently it is configured to view each of the attached IDE drives as individual drives. None of the cards RAID features are presently beeing used. Network is 100MB/s Switched Ethernet. Network cables and connects have been tested and verified. Software Environment: Very basic/vanilla Debian system install (Sid branch). Software package is rsync. Problem Description: When transfering many sometimes large files (3Gig in some cases)for backup purposes using rsync either via an ssh shell or rsync server the I20 drivers cause a kernel panic. The system seems to report increases in queue depth, shortly afterward the system completely hangs indicating a kernel panic. A kernel panic is by definition a kernel bug, not an application bug. Good luck! :-) p.s. kernel bug reports ought to say what kernel you're using. Steps to reproduce: Transfer files using rsync. Last specific command issued at prompt that reproduced this error was: rsync --bwlimit=2048 -vv -r -e ssh --delete --exclude lost+found rsync://[EMAIL PROTECTED]:873:/bu/area1/blue/* /bu/area1/blue This error does not seem to occour when transferrring the same file set using cp over nfs or scp. However, this does happen using rsync over nfs. -- Martin signature.asc Description: Digital signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: A question about rsync
On 31 May 2004, Guo jing [EMAIL PROTECTED] wrote: hello, I am a student in China.I like the linux and usually use the rsync to backup my documents. Last week when I use it,I find a question I want to discuss with you. The condition is like this: The source file that I want to rsync to another computer is 129M before I start the rsync. During the running of the rsync,the file was changed and became to about 50M, then the rsync ended. When I view the destination, I found that the file was 129M. And there were some contents of the files added when the rsync was running. After that, I do some tests about the rsync: 1. After I start the rsync to backup a file, I delete the file during the rsync is running, I found the file can been backuped normally. 2. While the rsync is backuping a file name sourfile (50M), I add some content by the command cat addfile sourfile to enlarge the file to 100M. After the rsync finished.I found the file is still 50M. The question is that , how the rsync copy a file to another computer at the first time ? My attitude is that it remenbers the physical blocks the file used when the rsync scaned. Then ,rsync will send the blocks to the destination no matter if the file or the block has changed. So, is that right?? Who can tell me how the rsync decide which contents should to send to the destiation? I'm not sure I understand the question, sorry. If you change a file while it is being copied by rsync you may end up with undefined results on the other end. There is not much that can be done about this without os-level version control. Sorry, my English is very poor. Thanks for your read and answer!! _ MSN Messenger: http://messenger.msn.com/cn -- Martin signature.asc Description: Digital signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug reporting
On Tue, 1 Jun 2004, Martin Pool wrote: On 1 Jun 2004, John Summerfield [EMAIL PROTECTED] wrote: The jitterbug link on http://rsync.samba.org/nobugs.html no longer works. I suggest it either be fixed or removed. Thanks, fixed. You make bug-reporting needlessly difficult, I think. I dislike the need to subscribe to a mailing-list and potentially receive lots of email that doesn't interest me. I have plenty of other email to keep me amused. I don't think you need to subscribe to post. I put the address directly on the nobugs page to make it easier for people to write to it. Did you have any other suggestions about how to make it better? Well, make it look like it's not a list? Most lists don't allow non-members to post, it's too easy to assume that this is the same. The reason we took Jitterbug and faq-o-matic down is that people seemed to get help more promptly when they wrote to the list. Cool. The way it was as confusing to noone's benefit. What I wanted most to do is ensure you know about rsyncx and consider working with the authors to create a unified product that supports resource forks when built on OSX. See http://www.macosxlabs.org./rsyncx/rsyncx.html Their CVS repository is at http://www.opendarwin.org It seems a shame to have two projects where one will do. Well, sometimes there are reasons not to glom everything into one big program. That may be so, but I don't think this is one of them. rsyncx is bound to have all rsync's bugs. Qute possibly some will be fixed first there, and it may be they will add valuable new features. One that I think would be nice is an automatic retry: I keep getting timeouts transferring between two ADSL accounts and I've not decide whether to blame Arachnet (the ISP at both ends) or rsync or even to suppose I'm expecting too much. wget has an option to automatically retry after timeouts, and it would certainly be useful to me if rsync did too. I may well decide to implement that myself: if I do, it would probably be in rsyncx if I decide that is where it would best suit me. Enough. I'm typing on a remote machine over an overtaxed dialup line:-( heers John -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: I20 Drivers Crash system when used with Rsync
Thanks Martin: Sorry about the kernel omission: 2.6.6 The bugzilla application setup for the kernel had a seperate field for the kernel version and I didn't add this information to the message I sent tot he list.. The bugzilla application assigned my report bug number: 2801 Dennis Martin Pool wrote: On 30 May 2004, Dennis R. Gesker [EMAIL PROTECTED] wrote: Note: I don't know if this is a problem withe I20 drivers or Rsync so I'm submitting to both the Kernel Bugzilla and the Rsync mailing list. I couldn't find a bugzilla for Rsync. I hope this was the correct way to submit this issue. Distribution: Debian Hardware Environment: Intel 850MV Mother board, Pentium 4 processor, 1Gig of RAM, Adaptec 2400A RAID Controler. Both the motherboard and Controller card have the most recent BIOS/firmware installed. The Adaptec card is capable of RAID configuration but currently it is configured to view each of the attached IDE drives as individual drives. None of the cards RAID features are presently beeing used. Network is 100MB/s Switched Ethernet. Network cables and connects have been tested and verified. Software Environment: Very basic/vanilla Debian system install (Sid branch). Software package is rsync. Problem Description: When transfering many sometimes large files (3Gig in some cases)for backup purposes using rsync either via an ssh shell or rsync server the I20 drivers cause a kernel panic. The system seems to report increases in queue depth, shortly afterward the system completely hangs indicating a kernel panic. A kernel panic is by definition a kernel bug, not an application bug. Good luck! :-) p.s. kernel bug reports ought to say what kernel you're using. Steps to reproduce: Transfer files using rsync. Last specific command issued at prompt that reproduced this error was: rsync --bwlimit=2048 -vv -r -e ssh --delete --exclude lost+found rsync://[EMAIL PROTECTED]:873:/bu/area1/blue/* /bu/area1/blue This error does not seem to occour when transferrring the same file set using cp over nfs or scp. However, this does happen using rsync over nfs. -- Dennis R. Gesker Manager -- Special Projects email: [EMAIL PROTECTED] Alamon Telco, Inc. 315 West Idaho Kalispell, MT 59901 email: [EMAIL PROTECTED] web: http://www.alamon.com phone: (800) 252-8838 phone: (406) 752-8838 fax: (800) 468-2645 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Unexplained error (code 24)
Hi all, While trying to mirror a filesystem from one machine to another (for backup purposes) I get the following error: [EMAIL PROTECTED] root]# /usr/bin/rsync -qavxzC --delete chandler:/var/ /bigdisk/backup/chandler/dev-md5-var/ [EMAIL PROTECTED]'s password: rsync error: unexplained error (code 24) at main.c(1045) Does this mean anything to anybody? Regards, Graham -- -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
include directory and all files under
Sorry, but it seems everytime I setup a different rsync operation (client only) I end up here unraveling the include/exclude stuff I need. The documentation is quite good but I guess the subject is just quite a bit to chew. To cut to the chase: I want to sync up my own snapshot repository of openbsd. The basic install files. They reside at: rsync://carroll.cac.psu.edu/openbsd/snapshots/i386/* However, there is also a `ports' source file that I want to include that resides at: rsync://carroll.cac.psu.edu/openbsd/snapshots/ports.tar.gz One level up from the other stuff. At that level, there are lots of different architectures etc I don't need so want to exclude. I'm trying this approach rsync [...] (various flags skipped) --include-from=./include --exclude=* rsync://carroll.cac.psu.edu/openbsd/snapshots/ /some/destination/ ./include looks like: cat include: ports.tar.gz i386 That gets the ports.tar.gz and and empty directory named i386 Trying: ports.tar.gz i386/* Doesn't get i386 or anything under it: Ditto for this: ports.tar.gz rsync://carroll.cac.psu.edu/openbsd/snapshots/i386/* or ports.tar.gz */i386/* So how is something like this done? And what do I need to know to be able to figure it out next time? -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug reporting
On Tue, 1 Jun 2004, John wrote: That may be so, but I don't think this is one of them. rsyncx is bound to have all rsync's bugs. Qute possibly some will be fixed first there, and it may be they will add valuable new features. One that I think would be nice is an automatic retry: I keep getting timeouts transferring between two ADSL accounts and I've not decide whether to blame Arachnet (the ISP at both ends) or rsync or even to suppose I'm expecting too much. That used to happen to me. Upgrading the firmware in my network hardware fixed the problem. wget has an option to automatically retry after timeouts, and it would certainly be useful to me if rsync did too. I may well decide to implement that myself: if I do, it would probably be in rsyncx if I decide that is where it would best suit me. The ideal place would be in ssh. Enough. I'm typing on a remote machine over an overtaxed dialup line:-( heers John -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug reporting
On Tue, 1 Jun 2004, Paul Haas wrote: On Tue, 1 Jun 2004, John wrote: That may be so, but I don't think this is one of them. rsyncx is bound to have all rsync's bugs. Qute possibly some will be fixed first there, and it may be they will add valuable new features. One that I think would be nice is an automatic retry: I keep getting timeouts transferring between two ADSL accounts and I've not decide whether to blame Arachnet (the ISP at both ends) or rsync or even to suppose I'm expecting too much. That used to happen to me. Upgrading the firmware in my network hardware fixed the problem. The Linux boxes? The Billion? It's got the latest already, at least as at a couple of weeks ago. The DSL-300? The ISP's side? I suspect that's where the problem is. We're not entirely happy with the ISP. wget has an option to automatically retry after timeouts, and it would certainly be useful to me if rsync did too. I may well decide to implement that myself: if I do, it would probably be in rsyncx if I decide that is where it would best suit me. The ideal place would be in ssh. ssh doesn't know enough, except maybe when the connexion dies. If it's rsync at the other end, to ssh it's just another program and it's ended. Possibly rsync can use ssh to forward a local port chosen the same way ftp chooses a port fo active ftp. Then local rsync opens a connexion to 127.0.0.1:port at the local end, and ssh forwards the stream to rsync running as a daemon (on another randomly-chosen port? a user-specified port?) a the destination. I've identified a couple of failures were due to the ADSL connexion dropping. The way of my life- Thunderbird's hung while I was checking email and writing two emails:-(( Enough. I'm typing on a remote machine over an overtaxed dialup line:-( heers John -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug reporting
John wrote: Possibly rsync can use ssh to forward a local port chosen the same way ftp chooses a port fo active ftp. Then local rsync opens a connexion to 127.0.0.1:port at the local end, and ssh forwards the stream to rsync running as a daemon (on another randomly-chosen port? a user-specified port?) a the destination. Bear in mind that this way you'll be encrypting the entire session instead of just the Please start rsyncd for me bit, so you'll likely suffer a speed hit depending on how powerful the two machines are. John also wrote... I keep getting timeouts transferring between two ADSL accounts and I've not decide whether to blame Arachnet (the ISP at both ends) or rsync or even to suppose I'm expecting too much. As an aisde, I've had a problem similar to the one you describe with an unrelated service. Turns out the NAT table was dropping entries that went unused for 5 minutes, which meant the firewall started blocking return traffic as it appeared to be unrelated to any outgoing connection. I don't know if this would ever be an issue with rsync but it might be worth a look. Terry. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Unexplained error (code 24)
/home/cnwt99/rsync-2.6.2grep ' 24 ' errcode.h #define RERR_VANISHED 24 /* file(s) vanished on sender side */ [EMAIL PROTECTED] /home/cnwt99/rsync-2.6.2 http://www.m-w.com/cgi-bin/dictionary?book=Dictionaryva=vanishedx=0y=0 Tim Conway Unix System Administration Contractor - IBM Global Services desk:3032734776 [EMAIL PROTECTED] Graham Leggett [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 06/01/2004 09:26 AM To [EMAIL PROTECTED] cc Subject Unexplained error (code 24) Hi all, While trying to mirror a filesystem from one machine to another (for backup purposes) I get the following error: [EMAIL PROTECTED] root]# /usr/bin/rsync -qavxzC --delete chandler:/var/ /bigdisk/backup/chandler/dev-md5-var/ [EMAIL PROTECTED]'s password: rsync error: unexplained error (code 24) at main.c(1045) Does this mean anything to anybody? Regards, Graham -- -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: (no subject)
Run it this way: /usr/local/bin/rsync -aHnuvvv serverX:/ / --exclude-from=/rsync.exclude --rsync-path=/usr/local/bin/rsync --ignore-existing 21 |tee /tmp/rsync.debug The extra verbosity will show you what transport you're using, and other problems. It could be that you're sshing in, and the authorized_keys file on serverX has command restrictions, for instance, so it just throws you out as soon as rsync invokes the remote. Combining stdout and stderr lets you see how they fit together. tee just lets you watch as it goes. Good luck. Tim Conway Unix System Administration Contractor - IBM Global Services desk:3032734776 [EMAIL PROTECTED] From the local machine the following command is executed: /usr/local/bin/rsync -aHnuv serverX:/ / --exclude-from=/rsync.exclude --rsync-path=/usr/local/bin/rsync --ignore-existing /var/tmp/rsync.stdout 2 /var/tmp/rsync.stderr I have never used the rsync command. The above command was used by a former sysadmin to synchronize two servers. However when I ran the command it did not produce any dry -run output (n option). A check of the /var/tmp/rsync.stderr shows the following output ws produced: rsync: connection unexpectedly closed (0 bytes read so far); rsync error: error in rsync protocl data stream (code 12) at io.c(165). Based on the above syntax, what was included/excluded that made this not work, and what is the correct syntax to sync the local machine to the remote? In advance, thanks for the help. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync for program loading on embedded platforms
On Tue, 2004-06-01 at 17:44, Greger Cronquist wrote: Hi, Has anyone considered using the rsync algorithm for loading programs on embedded platforms? I.e. with low cpu power (say, 16-25 MHz ARM7) and low memory requirements (programs are 128 KB, and the rsync algorithm I've been doing some work on librsync. As I come from an embedded background, I've always been conscious of embedded requirements as I worked on it. must use little ROM and RAM). Or is it just not worth the effort? The algorithm sacrifices CPU to minimise data transfer when updating it. For it to be worth it, you must already have similar data that needs updating, and data transfer must be expensive relative to CPU. Often with embedded systems you are loading programs from scratch, so there is nothing to update. Even if you are updating programs, compiled binaries are often very different for only minor source changes. It would be worth analysing your data to see if there are more application specific ways to minimise updates (like partitioning data into discrete parts that can be updated independently). I guess you'd want to at least modify the checksum calculations and use shorter and faster checksums (as the data to be transferred is smaller than on workstations), and you'd need that file overwrite patch that's been floating around. The checksum size you can use is a function of the data size, block size, and your acceptable failure rate. There are threads where this has been analysed and the latest rsync incorporates dynamic checksum and block sizing based on that discussion. librsync doesn't have it yet, but it could easily be added. I'd be interested to hear of anyone using or contemplating the rsync algorithm on embedded systems. -- Donovan Baarda [EMAIL PROTECTED] http://minkirri.apana.org.au/~abo/ -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: include directory and all files under
Tim Conway wrote: I personally have never used include/excludes, but I remember reading the man page a few years ago, and though it was very well written, I've never had a scenario where they were appropriate. From your specification, I don't think you do either. Simplify. the main use i've found for them is if you have tons and tons of directories, but similarly named files in those directories that need syncing. for instance i've got a web site with thousands of product directories, and there might be a file named review.html in all those directories. in that case i run the following include/exclude file: + / + review*.html - *.html - *.php -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsyncweb
Date: Tue Jun 1 09:08:29 2004 Author: mbp Update of /data/cvs/rsyncweb In directory dp.samba.org:/tmp/cvs-serv5025 Modified Files: nobugs.html Log Message: Clean up mention of mailing list. Revisions: nobugs.html 1.8 = 1.9 http://www.samba.org/cgi-bin/cvsweb/rsyncweb/nobugs.html.diff?r1=1.8r2=1.9 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsyncweb
Date: Tue Jun 1 16:58:07 2004 Author: wayned Update of /data/cvs/rsyncweb In directory dp.samba.org:/tmp/cvs-serv15655 Modified Files: header.html Log Message: Changed nobugs.html to bugzilla.html. Revisions: header.html 1.15 = 1.16 http://www.samba.org/cgi-bin/cvsweb/rsyncweb/header.html.diff?r1=1.15r2=1.16 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsyncweb
Date: Tue Jun 1 16:58:28 2004 Author: wayned Update of /data/cvs/rsyncweb In directory dp.samba.org:/tmp/cvs-serv15809 Added Files: bugzilla.html Log Message: First version of new bug-reporting web page. Revisions: bugzilla.html NONE = 1.1 http://www.samba.org/cgi-bin/cvsweb/rsyncweb/bugzilla.html?rev=1.1 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsyncweb
Date: Tue Jun 1 17:00:22 2004 Author: wayned Update of /data/cvs/rsyncweb In directory dp.samba.org:/tmp/cvs-serv17146 Removed Files: nobugs.html Log Message: Superseded by bugzilla.html. Revisions: nobugs.html 1.9 = NONE http://www.samba.org/cgi-bin/cvsweb/rsyncweb/nobugs.html?rev=1.9 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsyncweb
Date: Tue Jun 1 20:20:58 2004 Author: wayned Update of /data/cvs/rsyncweb In directory dp.samba.org:/tmp/cvs-serv32592 Modified Files: bugzilla.html Log Message: Improved one sentence. Revisions: bugzilla.html 1.2 = 1.3 http://www.samba.org/cgi-bin/cvsweb/rsyncweb/bugzilla.html.diff?r1=1.2r2=1.3 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
