RE: micro rsync
I posted something on the dd-wrt forum with no response but thanks for the tip about going to openwrt. But for some sort of answer here, what are the bare minimum files necessary for an rsync server with ssh? _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Paul Slootman Sent: Tuesday, October 30, 2007 12:14 PM To: rsync@lists.samba.org Subject: Re: micro rsync On Tue 30 Oct 2007, stevezemlicka wrote: Just wondering if anybody has thought about this. I would like to attempt to setup a router with dd-wrt and a NAS device as a home backup system without a computer. The processor is 200+MHz and I can have a maximum of 6MB on the device and a mounted samba storage volume. It may also be possible to have a USB hard drive in use instead of NAS. Is this possible. Thanks. I expect you may get more suitable answers on an openwrt list or forum, probably... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
period in folder name
I'm having a terrible time with one particular customer using rsync. Every once in awhile, they create a folder like Jon Const. Rsync seems to have a terrible time with the period in the folder name. After the backup runs and fails due to this folder, I cannot simply rename the folder. I have to go into cmd and rename the folder using the short filename. Has anybody run into this issue? Is there a way to get around it? I don't just want to rsync the short filenames because I want to keep everything as it is on the rsync client. Any suggestions would be appreciated. BTW, yes I've told them not to do it but they just don't seem to get it. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail mailto:[EMAIL PROTECTED] [EMAIL PROTECTED] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: period in folder name
XP Pro on both ends. This is the error in the log building file list ... file has vanished: /cygdrive/F/Share/CP Art/M-P/M/McCullough Const. done IO error encountered -- skipping file deletion Once I go into cmd and rename the folder via the short name to McCullough Const, it works just fine. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Thursday, October 11, 2007 10:15 PM To: Stephen Zemlicka Cc: rsync@lists.samba.org Subject: Re: period in folder name On 10/11/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: I'm having a terrible time with one particular customer using rsync. Every once in awhile, they create a folder like Jon Const. Rsync seems to have a terrible time with the period in the folder name. After the backup runs and fails due to this folder, I cannot simply rename the folder. I have to go into cmd and rename the folder using the short filename. Has anybody run into this issue? Is there a way to get around it? I don't just want to rsync the short filenames because I want to keep everything as it is on the rsync client. Any suggestions would be appreciated. Which OS is running on each side of the copy? Exactly how does the backup fail? This is likely to be a Windows or Cygwin idiosyncrasy. On my computer, Windows seems to remove trailing dots and spaces from every filename passed to a system call, so I can't create a folder whose name ends in a period. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: period in folder name
I'm not sure which but either an XP machine or a OSX machine is creating those folders. I'll check out the script, it shouldn't be too hard to do. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Thursday, October 11, 2007 11:02 PM To: Stephen Zemlicka Cc: rsync@lists.samba.org Subject: Re: period in folder name On 10/11/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: XP Pro on both ends. This is the error in the log building file list ... file has vanished: /cygdrive/F/Share/CP Art/M-P/M/McCullough Const. done IO error encountered -- skipping file deletion I imagine that once a period gets into the filename, readdir returns the name with the period, but when rsync attempts to stat the directory using that name, Windows strips off the period, causing the call to miss the actual file. What I'm wondering is how the period got there in the first place since Windows seems to strip periods when files are created (at least in my test on Windows Vista). Is the source on a mapped drive backed by a Linux machine, and are customers creating files directly on the Linux machine? In that case, Linux would accept the period and send it to Windows clients listing the mapped drive. In any case, there are two things you might try: (1) If the source is on a Linux machine, run the sending side of rsync on that machine (rather than over the mapped drive) so rsync can access the period-ending files correctly. Keep in mind that if rsync writes the files to a Windows destination, the periods will get stripped on the destination. (2) Before each backup, run a script that scans the source for period-ending files and renames them. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync error
I'm getting the attached error suddenly. I am using deltacopy for the ease
of setup and e-mail notifications. It is from a local disk to a USB drive.
I believe it's using rsync 2.6.6
_
Stephen Zemlicka
Integrated Computer Technologies
PH. 608-558-5926
E-Mail mailto:[EMAIL PROTECTED] [EMAIL PROTECTED]
Executing: rsync.exe -v -rlt --delete /cygdrive/C/Documents and
Settings/User/Local Settings/Application
Data/Identities/{DFF16927-88E6-4EAA-A097-460B7E65289B}/Microsoft/Outlook
Express/ localhost::Backup/Outlook Express/
building file list ...
done
./
Deleted Items.dbx
rsync: writefd_unbuffered failed to write 4 bytes: phase unknown [sender]:
Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at
/home/lapo/packaging/tmp/rsync-2.6.6/io.c(584)
Error in rsync protocol data stream
Rsync.exe returned an error. Will try again. This is retry number 1 of 5
Executing: rsync.exe -v -rlt --delete /cygdrive/C/Documents and
Settings/User/Local Settings/Application
Data/Identities/{DFF16927-88E6-4EAA-A097-460B7E65289B}/Microsoft/Outlook
Express/ localhost::Backup/Outlook Express/
building file list ...
done
./
Deleted Items.dbx
rsync: writefd_unbuffered failed to write 4 bytes: phase unknown [sender]:
Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at
/home/lapo/packaging/tmp/rsync-2.6.6/io.c(584)
Error in rsync protocol data stream
Rsync.exe returned an error. Will try again. This is retry number 2 of 5
Executing: rsync.exe -v -rlt --delete /cygdrive/C/Documents and
Settings/User/Local Settings/Application
Data/Identities/{DFF16927-88E6-4EAA-A097-460B7E65289B}/Microsoft/Outlook
Express/ localhost::Backup/Outlook Express/
building file list ...
done
./
Deleted Items.dbx
rsync: writefd_unbuffered failed to write 4 bytes: phase unknown [sender]:
Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at
/home/lapo/packaging/tmp/rsync-2.6.6/io.c(584)
Error in rsync protocol data stream
Rsync.exe returned an error. Will try again. This is retry number 3 of 5
Executing: rsync.exe -v -rlt --delete /cygdrive/C/Documents and
Settings/User/Local Settings/Application
Data/Identities/{DFF16927-88E6-4EAA-A097-460B7E65289B}/Microsoft/Outlook
Express/ localhost::Backup/Outlook Express/
building file list ...
done
./
Deleted Items.dbx
rsync: writefd_unbuffered failed to write 4 bytes: phase unknown [sender]:
Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at
/home/lapo/packaging/tmp/rsync-2.6.6/io.c(584)
Error in rsync protocol data stream
Rsync.exe returned an error. Will try again. This is retry number 4 of 5
Executing: rsync.exe -v -rlt --delete /cygdrive/C/Documents and
Settings/User/Local Settings/Application
Data/Identities/{DFF16927-88E6-4EAA-A097-460B7E65289B}/Microsoft/Outlook
Express/ localhost::Backup/Outlook Express/
building file list ...
done
./
Deleted Items.dbx
rsync: writefd_unbuffered failed to write 4 bytes: phase unknown [sender]:
Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at
/home/lapo/packaging/tmp/rsync-2.6.6/io.c(584)
Error in rsync protocol data stream
Executing: rsync.exe -v -rlt --delete /cygdrive/C/Documents and
Settings/User/Application Data/Microsoft/Address Book/
localhost::Backup/Address Book/
building file list ...
done
sent 66 bytes received 16 bytes 164.00 bytes/sec
total size is 719936 speedup is 8779.71
Executing: rsync.exe -v -rlt --delete /cygdrive/C/GRIP32/
localhost::Backup/GRIP32/
building file list ...
done
Pbd/
Pbd/Events.dat
rsync: writefd_unbuffered failed to write 4 bytes: phase unknown [sender]:
Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at
/home/lapo/packaging/tmp/rsync-2.6.6/io.c(584)
Error in rsync protocol data stream
Rsync.exe returned an error. Will try again. This is retry number 1 of 5
Executing: rsync.exe -v -rlt --delete /cygdrive/C/GRIP32/
localhost::Backup/GRIP32/
building file list ...
done
Pbd/
Pbd/Events.dat
rsync: writefd_unbuffered failed to write 4 bytes: phase unknown [sender]:
Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at
/home/lapo/packaging/tmp/rsync-2.6.6/io.c(584)
Error in rsync protocol data stream
Rsync.exe returned an error. Will try again. This is retry number 2 of 5
Executing: rsync.exe -v -rlt --delete /cygdrive/C/GRIP32/
localhost::Backup/GRIP32/
building file list ...
done
Pbd
RE: rsync error
I ran it again with no changes and it worked. Any ideas why it didn't run
the few times I tried it earlier.
Additionally, this backup contains about 100GB of photos. It seems to
backup each file every backup. Any ideas? If I run the -c, it doesn't
backup each file but it takes forever (over 1hr) to run.
_
Stephen Zemlicka
Integrated Computer Technologies
PH. 608-558-5926
E-Mail [EMAIL PROTECTED]
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt
McCutchen
Sent: Sunday, October 07, 2007 2:36 PM
To: Stephen Zemlicka
Cc: rsync@lists.samba.org
Subject: Re: rsync error
On 10/7/07, Stephen Zemlicka [EMAIL PROTECTED] wrote:
I'm getting the attached error suddenly. I am using deltacopy for the
ease
of setup and e-mail notifications. It is from a local disk to a USB
drive.
I believe it's using rsync 2.6.6
Executing: rsync.exe -v -rlt --delete /cygdrive/C/Documents and
Settings/User/Local Settings/Application
Data/Identities/{DFF16927-88E6-4EAA-A097-460B7E65289B}/Microsoft/Outlook
Express/ localhost::Backup/Outlook Express/
rsync: writefd_unbuffered failed to write 4 bytes: phase unknown
[sender]: Connection reset by peer (104)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at
/home/lapo/packaging/tmp/rsync-2.6.6/io.c(584)
Something is interfering with the connection between the rsync client
and the DeltaCopy-managed daemon to which it is pushing files, even
though the daemon is on localhost. Here are some things you could try
to get more information about what is causing the problem:
(1) Run rsync at -vvv verbosity level.
(2) Check the daemon's log file for any relevant messages.
(3) Run the rsync client directly at the command line instead of
through DeltaCopy. Perhaps strace rsync to see exactly what happens
before the failure.
Matt
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync error
Still seems to transfer all files. Funny thing is it seems to only do it for the pictures folder. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Sunday, October 07, 2007 8:25 PM To: Stephen Zemlicka Cc: rsync@lists.samba.org Subject: Re: rsync error On 10/7/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: Additionally, this backup contains about 100GB of photos. It seems to backup each file every backup. Any ideas? Try --modify-window=1 . Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync 3.0.0pre1 released
Can I compile this for cygwin or has someone already done so? _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Thursday, October 04, 2007 11:05 PM To: rsync@lists.samba.org Subject: Re: Rsync 3.0.0pre1 released On 10/4/07, Wayne Davison [EMAIL PROTECTED] wrote: I've just released rsync 3.0.0pre1, the first pre-release version for the upcoming 3.0.0 release. Hooray! I have built an RPM of rsync 3.0.0pre1 and posted it here: http://mattmccutchen.net/rsync/#rsync-packages This pre-release has finally given me the impetus to install and start using rsync 3.0.0* for everyday tasks, so I hope I will catch any bugs. Wayne, regarding the link on http://rsync.samba.org/download.html : I plan to continue to provide RPMs of release and pre-release versions of rsync, but of course they won't be RPMs patched with ACL support any more. Also, it would be nice if you updated the link to point to the URL above because the old URL will stop working sometime next year. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: estimating nightly diffs
To do that, you’d have to find out how much data is being added daily. I would think that would closely resemble your nightly diff copies. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail mailto:[EMAIL PROTECTED] [EMAIL PROTECTED] _ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Noam Birnbaum Sent: Tuesday, October 02, 2007 3:22 PM To: rsync@lists.samba.org Subject: estimating nightly diffs Hey all, Any suggestions on how to estimate how much data rsync would synchronize on average in a given installation? Assume that a full rsync has already run and the only data being updated is just the daily diffs. Thanks, noam Noam Birnbaum http://maccentricsolutions.com/ 877.luv.macs x89 Apple Certified Technical Coordinator Apple Certified Help Desk Specialist -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: why not a gui for rsync
There are times I use deltacopy locally. I have a couple clients backing up to removable hard disks with it. Just start the server service on the local machine. For the server's ip, type in 127.0.0.1 or localhost and you can backup locally with deltacopy. Deltacopy is the best gui for windows IMO. Plus, it supports e-mail notification which I utilize heavily. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Fabian Cenedese Sent: Monday, October 01, 2007 4:17 AM To: rsync@lists.samba.org Subject: Re: why not a gui for rsync At 00:04 01.10.2007 -0500, Robert wrote: snip Is there some works in the world trying to make a good gui for rsync and to let all people using this very good method to make a backup. If somebody is interesting to create this gui, let me know --Suresh Delta Copy: http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp It has a gui already. No need to reinvent the wheel. Delta Copy unfortunately only works in client/server mode. As I also need local mode this is no option (though it looks nice). So me too I still haven't found a simple, user-friendly GUI. bye Fabi -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: cygwin rsync windows to windows ACL problem
I have synced terabytes of data across DSL connections using robocopy with no problem. Robocopy is just as fast as rsync on whole files and I find it to be even faster on folders with large numbers of small files (though I hear the next release of rsync is supposed to speed that up). However, if you must use rsync, you can still use the robocopy that I specified earlier. Rsync would be responsible for all the file copying and robocopy would be responsible for only the attributes and permissions. Since rsync is a *nix utility, I doubt much effort will be put into compatibility with windows permissions. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 11:00 AM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Unfortunately I don't think it will work. We have two offices in the same corporate park connected by an unreliable radio backhaul, and several terabytes of data to sync daily, hence my desire to get rsync working :( PS: I will definitely check out robocopy for other uses, thank you :) On Monday 01 October 2007, Stephen Zemlicka wrote: Robocopy does copy only the new or changed files. However it does copy the whole files (similar to the -W switch in rsync). I use robocopy whenever I already have a secure connection (VPN) or when my backups don't include large (multi GB) database files. To see if robocopy will work by itself for you, add the -W switch and see if that's acceptable. It'll take a bit longer for changed files but will be just as fast for new files. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 10:50 AM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Does robocopy handle partial transfers, and backups of changed deleted files? My reason for wanting rsync is to maintain a mirror, and accomplish incremental backups via --backup --backup-dir=blah with rsync. On Monday 01 October 2007, Stephen Zemlicka wrote: I have also noticed that permissions aren't always copied under windows. I've heard you can use robocopy to do it. I have not because it hasn't been critical to my operation but supposedly, you can use something like ROBOCOPY source destination /XO /XN /XC /E /COPY:ATSOU After the backup script. This should not copy any files and just copy the file permissions and attributes. Let me know if that helps. BTW, robocopy is part of the windows 2003 resource kit. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 9:59 AM To: rsync@lists.samba.org Subject: cygwin rsync windows to windows ACL problem I am having problems using rsync on cygwin to properly copy the ownership of files from windows to windows. I am running: rsync -avPA //src_addr/some/path /local/dst/path/ Based on: http://www.mail-archive.com/rsync@lists.samba.org/msg18920.html and: http://www.nabble.com/Re:-I-need-rsync-+-acl-support-for-windows--t2462647. h tml ENV: CYGWIN=ntsec tty I am running this on Windows Server 2003 R2 SP2 (a DC), the SRC is a Windows XP SP2 domain member. I have tried patching the cygwin rsync source to enable ACL support, and had this confirmed by rsync --version This didn't work so I got rsync-acl-2.6.9.tar.bz2 from: http://mattmccutchen.net/myrsync/ and configured with --enable-acl-support This still does not work. Is what I am trying to do not possible, or am I just doing something wrong? Thanks for any input. -- - havoc -- - havoc -- - havoc -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: cygwin rsync windows to windows ACL problem
Yup, that's what I originally intended but I probably wasn't very clear. So basically you would add ROBOCOPY source destination /XO /XN /XC /E /COPY:ATSOU to run after the rsync finishes. How are you running rsync with no remote daemon running? Mapped drive? Why not run the remote daemon, I believe rsync can even initiate the remote daemon from the client side if you want. (hopefully my terminology is correct there) _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 11:28 AM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Intriguing, you're saying to use rsync to copy/sync the data, the use robocopy to only recursively set permissions? Robocopy can do this without transferring data? I haven't finished reading all about it yet, but have started with: http://en.wikipedia.org/wiki/Robocopy On Monday 01 October 2007, Stephen Zemlicka wrote: I have synced terabytes of data across DSL connections using robocopy with no problem. Robocopy is just as fast as rsync on whole files and I find it to be even faster on folders with large numbers of small files (though I hear the next release of rsync is supposed to speed that up). However, if you must use rsync, you can still use the robocopy that I specified earlier. Rsync would be responsible for all the file copying and robocopy would be responsible for only the attributes and permissions. Since rsync is a *nix utility, I doubt much effort will be put into compatibility with windows permissions. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 11:00 AM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Unfortunately I don't think it will work. We have two offices in the same corporate park connected by an unreliable radio backhaul, and several terabytes of data to sync daily, hence my desire to get rsync working :( PS: I will definitely check out robocopy for other uses, thank you :) On Monday 01 October 2007, Stephen Zemlicka wrote: Robocopy does copy only the new or changed files. However it does copy the whole files (similar to the -W switch in rsync). I use robocopy whenever I already have a secure connection (VPN) or when my backups don't include large (multi GB) database files. To see if robocopy will work by itself for you, add the -W switch and see if that's acceptable. It'll take a bit longer for changed files but will be just as fast for new files. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 10:50 AM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Does robocopy handle partial transfers, and backups of changed deleted files? My reason for wanting rsync is to maintain a mirror, and accomplish incremental backups via --backup --backup-dir=blah with rsync. On Monday 01 October 2007, Stephen Zemlicka wrote: I have also noticed that permissions aren't always copied under windows. I've heard you can use robocopy to do it. I have not because it hasn't been critical to my operation but supposedly, you can use something like ROBOCOPY source destination /XO /XN /XC /E /COPY:ATSOU After the backup script. This should not copy any files and just copy the file permissions and attributes. Let me know if that helps. BTW, robocopy is part of the windows 2003 resource kit. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 9:59 AM To: rsync@lists.samba.org Subject: cygwin rsync windows to windows ACL problem I am having problems using rsync on cygwin to properly copy the ownership of files from windows to windows. I am running: rsync -avPA //src_addr/some/path /local/dst/path/ Based on: http://www.mail-archive.com/rsync@lists.samba.org/msg18920.html and: http://www.nabble.com/Re:-I-need-rsync-+-acl-support-for-windows--t2462647. h tml ENV: CYGWIN=ntsec tty I am running this on Windows Server 2003 R2 SP2 (a DC), the SRC is a Windows XP SP2 domain member. I have tried patching the cygwin rsync source to enable ACL support, and had this confirmed
RE: why not a gui for rsync
What exactly are you looking for in monitoring it? The -v should give you a verbose output. If you want more details, I think you can specify multiple -v to give you a more verbose output. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Suresh Govindachar Sent: Monday, October 01, 2007 11:26 AM To: rsync@lists.samba.org Subject: RE: why not a gui for rsync Regarding the recommendation: Delta Copy: http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp It has a gui already. No need to reinvent the wheel. I went over the FAQ and the documentation for Delta Copy. Delta Copy does not provide any means to continually monitor an on-going rsync process. The most they do is: | Question: What type of logging is available in DeltaCopy | | Answer: Besides the logging options available in Rsync, | DeltaCopy provides two additional logs: | | deltac.log - available on DeltaCopy client | DeltaService.log - available on DeltaCopy server | | deltac.log file contains information about the DeltaCopy | client. It logs a message when a profile is run and | notifications emails are sent to the user. | | DeltaService.log is available where DeltaCopy server is | installed and contains messages when service is started or | stopped. | | It is important to know that rsync supports additional logging | that can be configured by specifying additional parameters | either on DeltaCopy client or server. Please refer to rsync | documentation for more information about logging options | available for rsync. | | When rsync.exe is executed in the back-ground by DeltaCopy, it | generates status messages which are sent to standard output. | DeltaCopy captures these messages and sends them via email. Observe that Delta Copy relies on rsync itself to provide insight into the status of the on-going transaction. So the question remains: In regard to output, rsync has the following features: a) -v option with various levels of verbosity b) --stats c) --log-file=foo d) messages rsync prints to stdout e) messages rsync prints to stderr. Where can one find an explanation of the format, content and interpretation of the outputs (a-e)? Thanks, --Suresh -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: cygwin rsync windows to windows ACL problem
I don't think so. Let's get terminology out of the way first. The computer sending data is the client and the computer storing the backups is the server. Rsync needs to read the whole file stored on the server to know what data the client needs to send. Since you are only running rsync on the client, the entire file needs to be transmitted from the server to the client, then read and analyzed. Then, a partial transfer may occur. So you can do a partial transfer but before that is possible, the client needs a full transfer from the server to see what it needs to send. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: havoc [mailto:[EMAIL PROTECTED] Sent: Monday, October 01, 2007 11:54 AM To: Stephen Zemlicka Subject: Re: cygwin rsync windows to windows ACL problem So even with a mapped drive --partial will have no effect? Is this cygwin/Windows specific behavior (as it does work on linux)? On Monday 01 October 2007, Stephen Zemlicka wrote: Remember, if you don't have the rsync daemon running on the remote computer, rsync will transfer whole files anyway. I'm working at getting around this issue with mapped drives but haven't had a lot of time recently to get it ironed out and fully tested. If you are not running rsync on the system you are backing up to, rsync will run exactly like robocopy (ie. with the -W option). _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 11:46 AM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Yes, mapped drive, kindof, it's a netbios default share, but essentially the same thing. I am trying to avoid any situation that requires special handling on the remote (client workstation) side, like additional software to install and maintain. On Monday 01 October 2007, Stephen Zemlicka wrote: Yup, that's what I originally intended but I probably wasn't very clear. So basically you would add ROBOCOPY source destination /XO /XN /XC /E /COPY:ATSOU to run after the rsync finishes. How are you running rsync with no remote daemon running? Mapped drive? Why not run the remote daemon, I believe rsync can even initiate the remote daemon from the client side if you want. (hopefully my terminology is correct there) _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 11:28 AM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Intriguing, you're saying to use rsync to copy/sync the data, the use robocopy to only recursively set permissions? Robocopy can do this without transferring data? I haven't finished reading all about it yet, but have started with: http://en.wikipedia.org/wiki/Robocopy On Monday 01 October 2007, Stephen Zemlicka wrote: I have synced terabytes of data across DSL connections using robocopy with no problem. Robocopy is just as fast as rsync on whole files and I find it to be even faster on folders with large numbers of small files (though I hear the next release of rsync is supposed to speed that up). However, if you must use rsync, you can still use the robocopy that I specified earlier. Rsync would be responsible for all the file copying and robocopy would be responsible for only the attributes and permissions. Since rsync is a *nix utility, I doubt much effort will be put into compatibility with windows permissions. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 11:00 AM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Unfortunately I don't think it will work. We have two offices in the same corporate park connected by an unreliable radio backhaul, and several terabytes of data to sync daily, hence my desire to get rsync working :( PS: I will definitely check out robocopy for other uses, thank you :) On Monday 01 October 2007, Stephen Zemlicka wrote: Robocopy does copy only the new or changed files. However it does copy the whole files (similar to the -W switch in rsync). I use robocopy whenever I already have a secure connection (VPN) or when my backups don't include large (multi GB) database files
RE: cygwin rsync windows to windows ACL problem
I'm not sure. If you can run the daemon on the server, you should. My situation doesn't allow me to do that but if yours does, you should. If you initiate the daemon from the client, then I don't believe there's anything extra running on the server except for during backups. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 12:06 PM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Would running an rsync daemon on the client side (using Matt's ACL enabled rsync) solve my ACL problem? On Monday 01 October 2007, havoc wrote: Yes, mapped drive, kindof, it's a netbios default share, but essentially the same thing. I am trying to avoid any situation that requires special handling on the remote (client workstation) side, like additional software to install and maintain. On Monday 01 October 2007, Stephen Zemlicka wrote: Yup, that's what I originally intended but I probably wasn't very clear. So basically you would add ROBOCOPY source destination /XO /XN /XC /E /COPY:ATSOU to run after the rsync finishes. How are you running rsync with no remote daemon running? Mapped drive? Why not run the remote daemon, I believe rsync can even initiate the remote daemon from the client side if you want. (hopefully my terminology is correct there) _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 11:28 AM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Intriguing, you're saying to use rsync to copy/sync the data, the use robocopy to only recursively set permissions? Robocopy can do this without transferring data? I haven't finished reading all about it yet, but have started with: http://en.wikipedia.org/wiki/Robocopy On Monday 01 October 2007, Stephen Zemlicka wrote: I have synced terabytes of data across DSL connections using robocopy with no problem. Robocopy is just as fast as rsync on whole files and I find it to be even faster on folders with large numbers of small files (though I hear the next release of rsync is supposed to speed that up). However, if you must use rsync, you can still use the robocopy that I specified earlier. Rsync would be responsible for all the file copying and robocopy would be responsible for only the attributes and permissions. Since rsync is a *nix utility, I doubt much effort will be put into compatibility with windows permissions. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 11:00 AM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Unfortunately I don't think it will work. We have two offices in the same corporate park connected by an unreliable radio backhaul, and several terabytes of data to sync daily, hence my desire to get rsync working :( PS: I will definitely check out robocopy for other uses, thank you :) On Monday 01 October 2007, Stephen Zemlicka wrote: Robocopy does copy only the new or changed files. However it does copy the whole files (similar to the -W switch in rsync). I use robocopy whenever I already have a secure connection (VPN) or when my backups don't include large (multi GB) database files. To see if robocopy will work by itself for you, add the -W switch and see if that's acceptable. It'll take a bit longer for changed files but will be just as fast for new files. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 10:50 AM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Does robocopy handle partial transfers, and backups of changed deleted files? My reason for wanting rsync is to maintain a mirror, and accomplish incremental backups via --backup --backup-dir=blah with rsync. On Monday 01 October 2007, Stephen Zemlicka wrote: I have also noticed that permissions aren't always copied under windows. I've heard you can use robocopy to do it. I have not because it hasn't been
RE: why not a gui for rsync
RTM _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: Suresh Govindachar [mailto:[EMAIL PROTECTED] Sent: Monday, October 01, 2007 12:06 PM To: 'Stephen Zemlicka'; rsync@lists.samba.org Subject: RE: why not a gui for rsync What exactly are you looking for in monitoring it? The -v should give you a verbose output. If you want more details, I think you can specify multiple -v to give you a more verbose output. As my question indicates, I know the above. | So the question remains: In regard to output, rsync has | the following features: | | a) -v option with various levels of verbosity | b) --stats | c) --log-file=foo | d) messages rsync prints to stdout | e) messages rsync prints to stderr. | | Where can one find an explanation of the format, content | and interpretation of the outputs (a-e)? What I do not know is how to interpret the output from rsync. To give some specific examples: 1) consider the message: 2007/09/13 16:50:52 [13688] receiving file list Isn't the list of files determined from the command line used to invoke rsync? So what this new list of files that are being received from where and by whom? 2) In the following three lines, 2007/09/13 17:44:29 [13688] f..t cmsstorage.lst 2007/09/13 17:50:53 [13688] f.st WINDOWS/SYSTEM/FFASTLOG.TXT 2007/09/13 17:50:53 [13688] f+++ WINDOWS/TEMP/CP1293.TMP what does the fstuff mean? 3) In the following, 2007/09/13 17:43:10 [13688] IO error encountered -- skipping file deletion for what transaction was the IO error encountered: while reading the source (which file exactly)?, while writing the destination (which file exactly)?, while talking from sender rsync process to receiver rsync process?, while talking from receiver rsync process to sender rsync process? 4) What are the consequences of the following skip? IO error encountered -- skipping file deletion How can I know determine whether or not the skipping had any bad effects? So the question is: Where can one find an explanation of the format, content and interpretation of the outputs (a-e)? --Suresh -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: why not a gui for rsync
Gotta love Google. http://samba.anu.edu.au/ftp/rsync/rsync.html http://optics.ph.unimelb.edu.au/help/rsync/rsync.html http://www.mediacollege.com/cgi-bin/man/page.cgi?topic=rsync http://www.ss64.com/bash/rsync.html _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: Suresh Govindachar [mailto:[EMAIL PROTECTED] Sent: Monday, October 01, 2007 12:42 PM To: 'Stephen Zemlicka'; rsync@lists.samba.org Subject: RE: why not a gui for rsync Stephen Zemlicka wrote: RTM Where is the Manual -- that was the question I asked: | Where can one find an explanation of the format, content | and interpretation of the outputs (a-e)? --Suresh -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: cygwin rsync windows to windows ACL problem
Check out cwRsync. Should allow you to install and use rsync without the entire cygwin installation. http://www.itefix.no/phpws/index.php?module=pagemasterPAGE_user_op=view_pag ePAGE_id=6MMN_position=23:23 _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 12:42 PM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Well, I'll try installing cygwin and an rsync daemon on one of the XP workstations to be backed up later tonight to see if having rsync on both ends solves the ACL issue. I just can't do it now as the workstations are in use. On Monday 01 October 2007, Stephen Zemlicka wrote: I'm not sure. If you can run the daemon on the server, you should. My situation doesn't allow me to do that but if yours does, you should. If you initiate the daemon from the client, then I don't believe there's anything extra running on the server except for during backups. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 12:06 PM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Would running an rsync daemon on the client side (using Matt's ACL enabled rsync) solve my ACL problem? On Monday 01 October 2007, havoc wrote: Yes, mapped drive, kindof, it's a netbios default share, but essentially the same thing. I am trying to avoid any situation that requires special handling on the remote (client workstation) side, like additional software to install and maintain. On Monday 01 October 2007, Stephen Zemlicka wrote: Yup, that's what I originally intended but I probably wasn't very clear. So basically you would add ROBOCOPY source destination /XO /XN /XC /E /COPY:ATSOU to run after the rsync finishes. How are you running rsync with no remote daemon running? Mapped drive? Why not run the remote daemon, I believe rsync can even initiate the remote daemon from the client side if you want. (hopefully my terminology is correct there) _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 11:28 AM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Intriguing, you're saying to use rsync to copy/sync the data, the use robocopy to only recursively set permissions? Robocopy can do this without transferring data? I haven't finished reading all about it yet, but have started with: http://en.wikipedia.org/wiki/Robocopy On Monday 01 October 2007, Stephen Zemlicka wrote: I have synced terabytes of data across DSL connections using robocopy with no problem. Robocopy is just as fast as rsync on whole files and I find it to be even faster on folders with large numbers of small files (though I hear the next release of rsync is supposed to speed that up). However, if you must use rsync, you can still use the robocopy that I specified earlier. Rsync would be responsible for all the file copying and robocopy would be responsible for only the attributes and permissions. Since rsync is a *nix utility, I doubt much effort will be put into compatibility with windows permissions. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of havoc Sent: Monday, October 01, 2007 11:00 AM To: rsync@lists.samba.org Subject: Re: cygwin rsync windows to windows ACL problem Unfortunately I don't think it will work. We have two offices in the same corporate park connected by an unreliable radio backhaul, and several terabytes of data to sync daily, hence my desire to get rsync working :( PS: I will definitely check out robocopy for other uses, thank you :) On Monday 01 October 2007, Stephen Zemlicka wrote: Robocopy does copy only the new or changed files. However it does copy the whole files (similar to the -W switch in rsync). I use robocopy whenever I already have a secure connection (VPN) or when my backups don't include large (multi GB) database files. To see if robocopy will work by itself for you, add the -W switch and see if that's
RE: why not a gui for rsync
Don't a where's the manual question buried in a page of text. Keep it short and concise and then people won't skim to try and help you without wasting their own time. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: Suresh Govindachar [mailto:[EMAIL PROTECTED] Sent: Monday, October 01, 2007 12:59 PM To: 'Stephen Zemlicka'; rsync@lists.samba.org Subject: RE: why not a gui for rsync In response to Stephen Zemlicka wrote: RTM Where is the Manual -- that was the question I asked: | Where can one find an explanation of the format, content | and interpretation of the outputs (a-e)? where (a-e) refer to: | So the question remains: In regard to output, rsync has the | following features: | | a) -v option with various levels of verbosity | b) --stats | c) --log-file=foo | d) messages rsync prints to stdout | e) messages rsync prints to stderr. | | Where can one find an explanation of the format, content | and interpretation of the outputs (a-e)? Stephen Zemlicka wrote: SZ Gotta love Google. SZ SZ http://samba.anu.edu.au/ftp/rsync/rsync.html SZ http://optics.ph.unimelb.edu.au/help/rsync/rsync.html SZ http://www.mediacollege.com/cgi-bin/man/page.cgi?topic=rsync SZ http://www.ss64.com/bash/rsync.html Have you actually looked at the above links? Do they in fact answer the question asked? They do not. Please do not blindly respond to posts -- your responses are just a waste of time. --Suresh -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: reducing file list bytes transferred
I think the next release is supposed to speed this up. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Peter Salameh Sent: Monday, October 01, 2007 5:42 PM To: rsync@lists.samba.org Subject: reducing file list bytes transferred Hello, This is my first posting to the rsync list. I mirror a database containing directories which contain a very large number of files (say 30,000), and sending the file list can often take longer than transferring the new files.(Rsync ends up sending nearly the same file list on every transfer, with only the addition of a few new files.) Has the rsync team considered an rsync option which would remember the last file list on both ends, and only send changes to the list? In my case this would reduce the number of files in the file list from 300,000 per day to under 1,000 per day. Might be a good addition to rsync-3.0. Thanks in advance. Peter Salameh [EMAIL PROTECTED] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: reducing file list bytes transferred
I thought it was supposed to do it more effeciently _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Monday, October 01, 2007 8:10 PM To: Stephen Zemlicka Cc: Peter Salameh; rsync@lists.samba.org Subject: Re: reducing file list bytes transferred On 10/1/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: I think the next release is supposed to speed this up. Are you thinking of incremental recursion? It interleaves file list and file data transmission but does not decrease the total amount of file-list data sent. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Mapped Drive
OK, let's say this is the first sync and every file is being transferred. The checksum for each of the files is cached on the local drive. Then, the next time you sync, it checks the checksum from the cache against the file to be copied. If it matches, it skips it. If it doesn't match, it just transfers just the difference. It then replaces the checksum of that transferred file to the cache. That way one could have a remote data store and not have to run rsync on the remote system. IE, you could have a mapped drive or FTP folder or S3 storage area that would all be rsyncable. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Sunday, September 30, 2007 11:54 AM To: Stephen Zemlicka Cc: rsync@lists.samba.org Subject: Re: Mapped Drive On 9/30/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: The problem is some files don't change in size. So I was hoping that the checksums could be cached. Perhaps I'm mistaken but I thought the checksum determined what actual blocks were transferred. I suppose it could be cached at either storage location. I still do not understand what you are proposing; please be more specific! Which checksums are you talking about: (1) the whole-file checksums of destination files used by the -c option to decide whether to transfer each file, or (2) the block checksums of destination files used by the delta-transfer algorithm to match blocks, or (3) something else? At what point in the process would the cache be read, and at what point would it be updated? Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Mapped Drive
The problem is some files don't change in size. So I was hoping that the checksums could be cached. Perhaps I'm mistaken but I thought the checksum determined what actual blocks were transferred. I suppose it could be cached at either storage location. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Saturday, September 29, 2007 5:03 PM To: Stephen Zemlicka Cc: rsync@lists.samba.org Subject: Re: Mapped Drive On 9/28/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: Is there a way to have rsync cache the checksums for something like this and would that help? I'm not sure exactly what you mean. You said you were using the -c (--checksum) option, which makes rsync decide whether to update each destination file by reading the file in full and comparing its MD4/MD5 checksum with that of the source file. Do you mean you want rsync to cache the checksums of the destination files? On which machine would the cache be? Anyway, if the issue is that you don't want rsync spending the bandwidth to read the destination files for the --checksum check, just remove -c and rsync will use the default size-and-mtime quick check. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Mapped Drive
Is there a way to have rsync cache the checksums for something like this and would that help? _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Monday, September 24, 2007 6:51 PM To: Stephen Zemlicka Cc: rsync@lists.samba.org Subject: Re: Mapped Drive On 9/24/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: I think I got it (with the help of a friend). You can, in fact, rsync to a mapped drive efficiently. You must include --no-whole-file. My 100MB file only transferred a few MB using that method and it opened up with the changes just fine. Now it works just like I want. It's not the quickest but definitely quicker than uploading hundreds of MB for each database. Of course --no-whole-file will make rsync report a smaller number of bytes sent, but is it actually reducing the amount of I/O to the mapped drive? If so, I find this really surprising. The one explanation I can think of is that the network filesystem has cached the file, so reading the basis file costs nothing and writing only the changed areas (due to --inplace) is a big saving. Is this the case? Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Mapped Drive
After timing it, I found you're right. Would this still help for online storage though as most have much greater download bandwidth than upload. So it would basically download the file with your faster download speeds, compare, then upload the changes with your slower upload speeds. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Monday, September 24, 2007 6:51 PM To: Stephen Zemlicka Cc: rsync@lists.samba.org Subject: Re: Mapped Drive On 9/24/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: I think I got it (with the help of a friend). You can, in fact, rsync to a mapped drive efficiently. You must include --no-whole-file. My 100MB file only transferred a few MB using that method and it opened up with the changes just fine. Now it works just like I want. It's not the quickest but definitely quicker than uploading hundreds of MB for each database. Of course --no-whole-file will make rsync report a smaller number of bytes sent, but is it actually reducing the amount of I/O to the mapped drive? If so, I find this really surprising. The one explanation I can think of is that the network filesystem has cached the file, so reading the basis file costs nothing and writing only the changed areas (due to --inplace) is a big saving. Is this the case? Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync Truecrypt
I thought I found the answer to my problem in the -no-whole-file. However this still took as long as the initial transfer but it only actually uploaded a small bit. However, my issue is that I cannot run rsync on the server. From my tests, rsync will efficiently transfer truecrypt volumes if you are running rsync correctly (ie. On both ends). Again, my problem stems from my mapped drive solution and the lack of rsync on the server. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail mailto:[EMAIL PROTECTED] [EMAIL PROTECTED] _ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Frank Hamersley Sent: Tuesday, September 25, 2007 7:25 AM To: rsync@lists.samba.org Subject: RE: Rsync Truecrypt Hmmm - I had expected that the truecrypt volume would exhibit far too much entropy for rsync to do an efficient difference based transferso I had planned to run against the mounted tc volume. I hadn't even planned to test the throughput! Should I reconsider? Cheers, Frank. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Stephen Zemlicka Sent: Saturday, 22 September 2007 5:49 AM To: rsync@lists.samba.org Subject: Rsync Truecrypt I have seen people say both that rsync can efficiently copy truecrypt volumes. I got it working once with deltacopy but as I'm trying to get it to work from the command line, I cannot. It just transfers the whole file. I am using -rvs -inplace switches. Does anybody have any suggestions? _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail mailto:[EMAIL PROTECTED] [EMAIL PROTECTED] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync and opened files
This doesn't help with linux but a gentleman named Elias added vss capabilities to rsync. However, the version is a bit old (I think 2.6 or around there). As soon as 3 is out, I may try to go through it line by line and add it into 3. Otherwise, check out microsoft's vss sdk. There's a bit of scripting but it should allow you to backup open files. Plus, services like sql and exchange that are vss aware. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Egoitz Aurrekoetxea Sent: Tuesday, September 25, 2007 8:19 AM To: rsync@lists.samba.org Subject: Rsync and opened files Hello, I'm trying to determine if rsync is a sure method of backing up servers (Linux and Windows) whose files are constantly being accesed and are not able to be stoped they're services for backing up purposes... I would use it over ssh for making incremental backups... in my tests seem to always have worked backing up from a debian server to the copy server that runs debian too... I'm using the next : OPTS=--force --ignore-errors --delete --backup --backup-dir=/home/ramattack/pruebas-rsync/$BACKUPDIRMES/$dia -avz rsync $OPTS $BDIR /home/ramattack/pruebas-rsync/$BACKUPDIRMES/imagen-copia BDIR is source I want to backup and /home/ramattack/pruebas-rsync... is the destination... could this copy correctly opened files? Normally I will use it for backing up linux machines normally... and the backup server will be of course a linux machine (debian machine). but how does it behave with linux machines? P.D. I have googled and searched over there but all posts I've find are old... and I wanted to have a recent answer. Thanks a lot mates!!! -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Mapped Drive
I am having trouble running rsync over a mapped drive. Basically it only copies whole files. I use the -rvcS switches. Any suggestions? TIA _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail mailto:[EMAIL PROTECTED] [EMAIL PROTECTED] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Mapped Drive
The reason I was trying this was because all I can do is map a drive. As of yet, I cannot modify the server like that. I tried the --inplace and it still did the whole file. Also, this will eventually be over the internet but right now I'm doing it locally (faster testing). I may take a peek at rdiff. Any experience with that? _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Monday, September 24, 2007 12:24 PM To: Stephen Zemlicka Cc: rsync@lists.samba.org Subject: Re: Mapped Drive On 9/24/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: What exactly is involved in the remote shell? Can this be done on a windows to windows backup? Do you have a link handy otherwise google to the rescue. Thank you very much. You set up an ssh server on the machine with the mapped drive, install an ssh client on the machine running rsync, and direct rsync to the mapped drive using a path of the form host:/cygdrive/C/path/to/mapped/drive/ (note the *single* colon). cwRsync ( http://itefix.no/cwrsync/ ) is a nice packaging containing rsync, an ssh client and server, and some glue to get the ssh server running in Windows; I recommend it. Note: you might run into the infamous remote-rsync-under-Cygwin hang bug, which is being tracked here: https://bugzilla.samba.org/show_bug.cgi?id=2208 Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Mapped Drive
I'm pushing. I thought rdiff could install just the changes to a dir and then when you restore, it applies the changes from the dir you specify. Is that incorrect? If rdiff does do that, I could write a gui or wrapper for it to make restores more user friendly. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Monday, September 24, 2007 12:46 PM To: Stephen Zemlicka Cc: rsync@lists.samba.org Subject: Re: Mapped Drive On 9/24/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: The reason I was trying this was because all I can do is map a drive. As of yet, I cannot modify the server like that. If the only access you have to the remote directory is to read or write it via a mapped drive, you can't hope to do any better than copying the whole file. All approaches to delta transfers rely on performing some kind of special computation on the remote machine. BTW, are you pushing files to the mapped drive or pulling files from it? Knowing that would make future discussion clearer. I may take a peek at rdiff. Any experience with that? rdiff is essentially a command-line interface to the three stages of the rsync delta-transfer algorithm. It won't work any better than rsync in this setting. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Mapped Drive
I think I got it (with the help of a friend). You can, in fact, rsync to a mapped drive efficiently. You must include --no-whole-file. My 100MB file only transferred a few MB using that method and it opened up with the changes just fine. Now it works just like I want. It's not the quickest but definitely quicker than uploading hundreds of MB for each database. Thanks you for the assistance. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matt McCutchen Sent: Monday, September 24, 2007 12:46 PM To: Stephen Zemlicka Cc: rsync@lists.samba.org Subject: Re: Mapped Drive On 9/24/07, Stephen Zemlicka [EMAIL PROTECTED] wrote: The reason I was trying this was because all I can do is map a drive. As of yet, I cannot modify the server like that. If the only access you have to the remote directory is to read or write it via a mapped drive, you can't hope to do any better than copying the whole file. All approaches to delta transfers rely on performing some kind of special computation on the remote machine. BTW, are you pushing files to the mapped drive or pulling files from it? Knowing that would make future discussion clearer. I may take a peek at rdiff. Any experience with that? rdiff is essentially a command-line interface to the three stages of the rsync delta-transfer algorithm. It won't work any better than rsync in this setting. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync Truecrypt
I have seen people say both that rsync can efficiently copy truecrypt volumes. I got it working once with deltacopy but as I'm trying to get it to work from the command line, I cannot. It just transfers the whole file. I am using -rvs -inplace switches. Does anybody have any suggestions? _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail mailto:[EMAIL PROTECTED] [EMAIL PROTECTED] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: File changed during save....
http://users.tkk.fi/~epenttil/rsync-vss/ _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Sunday, September 16, 2007 5:16 PM To: [EMAIL PROTECTED]; [EMAIL PROTECTED] Cc: rsync@lists.samba.org Subject: RE: File changed during save sounds interesting - are you speaking about a special rsync version or about this helper script: http://marc.info/?l=rsyncm=115822570129821w=2 ? -Ursprüngliche Nachricht- Von: Stephen Zemlicka [EMAIL PROTECTED] Gesendet: 16.09.07 19:43:54 An: 'roland' [EMAIL PROTECTED] CC: rsync@lists.samba.org Betreff: RE: File changed during save If you're on windows, someone wrote a vss patch for rsync. I haven't used it extensively though but it has worked for in-use outlook pst files so far. I plan on testing it with exchange and sql databases in the near future. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tony Abernethy Sent: Sunday, September 16, 2007 11:24 AM To: 'Matt McCutchen'; 'roland' Cc: rsync@lists.samba.org Subject: RE: File changed during save Matt McCutchen wrote: On 9/15/07, roland [EMAIL PROTECTED] wrote: what`s the rsync equivalent to this? how can i see which files changed while rsync was transferring them ? Handling of concurrent changes to source files is one of rsync's weaknesses. The rsync sender naively reads each source file from beginning to end and sends what it sees; it doesn't detect if source files change while being transferred. In many cases, the concurrent modification will leave the source file with an mtime different from the mtime rsync saw when it statted the file during file-list building (which gets set on the destination file), so a subsequent rsync run will fix the corrupted destination file. See this thread for more information: http://lists.samba.org/archive/rsync/2006-January/014534.html Matt -- Note that back-to-back rsyncs make the window of opportunity much much smaller for things to change during transit. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html _ Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! http://smartsurfer.web.de/?mc=100071distributionid=0066 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: File bit synchronization?
Rsync performs delta copies. This is by far the best feature of rsync IMHO. I have many backup solutions setup for a number of clients that perform offsite synchronization of multi GB sql, exchange, etc. databases in minutes over standard DSL connections. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Robert Fitzpatrick Sent: Monday, September 17, 2007 3:49 PM To: rsync Subject: File bit synchronization? Not sure if that is the correct terminology. I have an rsync setup of files between two Windows servers using cwRsync, which uses -apv options among --progress and --delete options. One part of the backup is to transfer a MSSQL backup file and I notice that after the initial transfer taking 20 minutes or more, subsequent daily transfers after is changes each night take only a minute or two max and this is the summary... Number of files: 1 Number of files transferred: 1 Total file size: 448610816 bytes Total transferred file size: 448610816 bytes Literal data: 3807632 bytes Matched data: 444803184 bytes File list size: 77 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 148389 Total bytes received: 3892615 Although the file size does not change very much day to day, the file is generated from MSSQL backup function every day and does change, and the destination date time stamp updates to match the source. If I wipe out the file on the destination and start again, several minutes to transfer the file. So, I assume there is some sort of 'bit synchronization' that only refreshes the file contents? This is a binary file and I did not know this was possible. I've been running rsync between unix/linux boxes for years doing backup and would love this to work with our PostgreSQL backups. I've tried the same options and choosing pg_dump format options of both 'c' and 'p' (plain text), but nothing works. It always transfers the entire file. I'm I seeing this correctly that rsync can do this? If so, what are the requirements to make it happen? Thanks for the help... -- Robert -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: File bit synchronization?
No idea why it's not. I usually use the -v -rlt -z --delete options. You could try the -c switch. I believe that will ignore the mod time and size and skip based on the checksum only. Though I would think you need a switch that does the opposite of that. Maybe someone else has an idea. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: Robert Fitzpatrick [mailto:[EMAIL PROTECTED] Sent: Monday, September 17, 2007 4:38 PM To: Stephen Zemlicka Cc: 'rsync' Subject: RE: File bit synchronization? On Mon, 2007-09-17 at 16:02 -0500, Stephen Zemlicka wrote: Rsync performs delta copies. This is by far the best feature of rsync IMHO. I have many backup solutions setup for a number of clients that perform offsite synchronization of multi GB sql, exchange, etc. databases in minutes over standard DSL connections. Great! Thanks. Why do you think my pgsql backup from pg_dump in postgresql 8.2.4 does not seem to be doing this? Do you mind if I ask what options you are using with rsync? -- Robert -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync Active Mirroring
Can rsync support active mirroring? I've used robocopy in the past and it has a -mon switch that can copy ever x changes or every x minutes. I'm mostly interested in the x changes as the x minutes could easily be scheduled. Can rsync keep the data encrypted rather than unencrypting on the other end? Let me explain better. I want to rsync to a removable disk but I want the data to be kept in the original file names and directory structure but I want the files to be encrypted so they cannot be opened without the key. I want to use the blowfish 448 encryption unless someone knows of a better. I have the patched version of rsync that is supposed to be vss aware or something like that. I've used it successfully to rsync open Outlook pst files without a problem. I want to start using it to backup open sql and exchange databases. I've heard of this vss aware copy having problems. Anybody have any experience with it or any additional info? Thank you very much for taking the time to read and help with these issues. I've done a bit of research and didn't find much so hopefully these haven't been covered too much before. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail mailto:[EMAIL PROTECTED] [EMAIL PROTECTED] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: File changed during save....
If you're on windows, someone wrote a vss patch for rsync. I haven't used it extensively though but it has worked for in-use outlook pst files so far. I plan on testing it with exchange and sql databases in the near future. _ Stephen Zemlicka Integrated Computer Technologies PH. 608-558-5926 E-Mail [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tony Abernethy Sent: Sunday, September 16, 2007 11:24 AM To: 'Matt McCutchen'; 'roland' Cc: rsync@lists.samba.org Subject: RE: File changed during save Matt McCutchen wrote: On 9/15/07, roland [EMAIL PROTECTED] wrote: what`s the rsync equivalent to this? how can i see which files changed while rsync was transferring them ? Handling of concurrent changes to source files is one of rsync's weaknesses. The rsync sender naively reads each source file from beginning to end and sends what it sees; it doesn't detect if source files change while being transferred. In many cases, the concurrent modification will leave the source file with an mtime different from the mtime rsync saw when it statted the file during file-list building (which gets set on the destination file), so a subsequent rsync run will fix the corrupted destination file. See this thread for more information: http://lists.samba.org/archive/rsync/2006-January/014534.html Matt -- Note that back-to-back rsyncs make the window of opportunity much much smaller for things to change during transit. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
