rsync with danish chars in filename
Hi, I have a Qnap NAS-219 used for storage, i have a Ubuntu 9.04 server for backup. When rsync between them filenames containing ø have problemes, the danish chars æ å does not have any problems eventhough its a special char like æ ø å. The log files state following: file has vanished: /M?deskabelon.doc where ? should be ø, and the file is stil at the system, so its not vanished. The charset on NAS is: en_US.UTF-8 And backup-server is: en_DK.UTF-8 I know there is a difference but in the starting wizard at the nas, it was the recommended charset used for danish language files. And the problem had been more understandable if the charset was like: The charset on NAS is: en_DK.UTF-8 And backup-server is: en_US.UTF-8 then the backup-server should not understand danish special chars like æ ø å. Hope someone can help me or give me a hint. Best regards Tomas -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync with danish chars in filename
Hi Tomas file has vanished: /M?deskabelon.doc I have had a similar problem, where the source file was not in the correct encoding (used on the source). So in my case, the filename on the source contained chars that was not properly encoded in the source machine's encoding. (This was a Windows server). I don't know if this is the same situation, you are in. Have you tried creating a new file on the source with a filename containing 'ø'? Med venlig hilsen/Kind regards Thomas Damgaard Nielsen http://thomasdamgaard.dk -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync with danish chars in filename
Hi Thomas (Brovst ? :) 2009/9/17 Thomas Damgaard thoma...@gmail.com: Hi Tomas file has vanished: /M?deskabelon.doc I have had a similar problem, where the source file was not in the correct encoding (used on the source). So in my case, the filename on the source contained chars that was not properly encoded in the source machine's encoding. (This was a Windows server). I don't know if this is the same situation, you are in. Have you tried creating a new file on the source with a filename containing 'ø'? Yes, have tried that with these yesterday: Created To folder: Testmappe æøå - systemname - computername and Testmappe - systemname - computername, and within those i have made 5 files: filename.txt filenameæ.txt filenameø.txt filenameå.txt filenameæøå.txt to se which chars fails. Only the one with ø fails, but the directories, does not fail. I have tried to make the to folders mentioned from 7 different systems, from Vista, XP, Win7, Ubuntu, FreeBSD etc. Med venlig hilsen/Kind regards Thomas Damgaard Nielsen http://thomasdamgaard.dk -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
quick check failed
Hi folks, We ran 2 mirror jobs using rsync -av remote:/from /to of appr. 700 GByte. Problem: On the second run about 10 or 15 files were copied again, even though we are very sure that there was no service running which could have changed the files on remote:/from or on /to. Esp. Samba and NFS were off. How comes? The remote host has rsync version 3.0.4. The localhost uses version 3.0.3. The file system was reiserfs on the remote side, and xfs on localhost. Any helpful comment would be highly appreciated Harri -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Limit rsync running time
Hi I'd like to rsync a large amount of data over a slow connection, but only during night hours. I couldn't find a parameter that limits the time that rsync is running, only the timeout on idle time. I guess the way to go would be to start rsync, get the process ID and kill the process later on. Has anybody already written a bash script that would do something like that? Are there other ways? I don't want to kill all rsync processes as there might be other syncs going on. Thanks bye Fabi -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Limit rsync running time
Hi, I have done this with a cron job, only the first time i had to transfer 15GB, but did it first time in the weekend, afterwards, in my situation is only a small amount of data that has to be transfered every night, so every day at 02.00 the rsync starts and stops automaticly within 1-3 hours. Best Regards Tomas 2009/9/17 Fabian Cenedese cened...@indel.ch: Hi I'd like to rsync a large amount of data over a slow connection, but only during night hours. I couldn't find a parameter that limits the time that rsync is running, only the timeout on idle time. I guess the way to go would be to start rsync, get the process ID and kill the process later on. Has anybody already written a bash script that would do something like that? Are there other ways? I don't want to kill all rsync processes as there might be other syncs going on. Thanks bye Fabi -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Limit rsync running time
On Thu 17 Sep 2009, Fabian Cenedese wrote: Has anybody already written a bash script that would do something like that? Are there other ways? I don't want to kill all rsync processes as there might be other syncs going on. There is for example a timeout package available in Debian: Package: timeout Description: run a command with a time limit timeout executes a command and imposes an elapsed time limit. When the time limit is reached, timeout sends a predefined signal to the target process. Homepage: http://www.porcupine.org/forensics/tct.html So use that: $ timeout 7200 rsync That will kill rsync after 2 hours (7200 seconds). Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Limit rsync running time
At 15:14 17.09.2009 +0200, Paul Slootman wrote: On Thu 17 Sep 2009, Fabian Cenedese wrote: Has anybody already written a bash script that would do something like that? Are there other ways? I don't want to kill all rsync processes as there might be other syncs going on. There is for example a timeout package available in Debian: Package: timeout Description: run a command with a time limit timeout executes a command and imposes an elapsed time limit. When the time limit is reached, timeout sends a predefined signal to the target process. Homepage: http://www.porcupine.org/forensics/tct.html That would be what I need. But I have rsync running on a NAS with BusyBox, so I'd need to cross-compile it. That's why I was looking for a bash script. If other people have the same need: I found a script on the net: http://twoday.tuwien.ac.at/jo/stories/320762/ Thanks bye Fabi -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 6741] New: 'deleting' messages show up in improper places
https://bugzilla.samba.org/show_bug.cgi?id=6741 Summary: 'deleting' messages show up in improper places Product: rsync Version: 3.0.5 Platform: x86 OS/Version: Linux Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: way...@samba.org ReportedBy: asch...@gmail.com QAContact: rsync...@samba.org The 'deleting' messages below are showing up in the middle of a transfer. It shows files being deleted and the progress bar appears to detail the deleting process. Usually rsync handles files in alphanumeric order, but there are still a number of files to transfer in the Dio directory. Dio/path/file.ext deleting Donovan/folder.jpg deleting Donovan/path/folder.jpg 5046272 51% 78.19kB/s0:01:01 -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 6741] 'deleting' messages show up in improper places
https://bugzilla.samba.org/show_bug.cgi?id=6741 --- Comment #1 from asch...@gmail.com 2009-09-17 09:24 CST --- a...@eee1:~$ rsync --version rsync version 3.0.5 protocol version 30 Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Limit rsync running time
Rsync has a 'time-limit' patch with following options: --stop-at=y-m-dTh:m Stop rsync at year-month-dayThour:minute --time-limit=MINS Stop rsync after MINS minutes have elapsed Tev Hi I'd like to rsync a large amount of data over a slow connection, but only during night hours. I couldn't find a parameter that limits the time that rsync is running, only the timeout on idle time. I guess the way to go would be to start rsync, get the process ID and kill the process later on. Has anybody already written a bash script that would do something like that? Are there other ways? I don't want to kill all rsync processes as there might be other syncs going on. Thanks bye Fabi -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync deletes target files after source drive failure
Hi, I'm making backups using rsnapshot via cron jobs. Recently the source drive died. rsync wasn't able to read the source data and subsequently deleted the target directory. Here is part from the log: . . . /usr/bin/rsync -avi --delete --numeric-ids --relative --delete-excluded \ --stats --exclude-from=/redgum-backup/emiddel/rsync-excludes \ --rsh=/usr/bin/ssh x...@yyy:/srcdir \ /targetdir/ receiving incremental file list . . . rsync: readdir(/data/1934-638.1344.04nov27): Input/output error (5) rsync: readdir(/data/1934-638.1344.04nov28): Input/output error (5) . . . deleting data/1934-638.1344.04nov27/visdata deleting data/1934-638.1344.04nov27/vartable deleting data/1934-638.1344.04nov27/leakage deleting data/1934-638.1344.04nov27/history deleting data/1934-638.1344.04nov27/header deleting data/1934-638.1344.04nov27/gains deleting data/1934-638.1344.04nov27/flags deleting data/1934-638.1344.04nov27/bandpass deleting data/1934-638.1344.04nov28/visdata deleting data/1934-638.1344.04nov28/vartable deleting data/1934-638.1344.04nov28/leakage deleting data/1934-638.1344.04nov28/history deleting data/1934-638.1344.04nov28/header deleting data/1934-638.1344.04nov28/gains deleting data/1934-638.1344.04nov28/flags . . . Here the two directories which couldn't be read have been deleted from the target directory (ie, the files contained in them were deleted, the directories did still exist, if I remember correctly). If I understand the man page correctly, rsync should not delete files from the target directory when an error occurs. This can be overridden with the --ignore-errors command line switch, but this switch has definitely not been used. This error appears to be very similar to what has been reported on two years ago (http://lists.samba.org/archive/rsync/2007-September/018569.html) and according to https://bugzilla.samba.org/show_bug.cgi?id=4979 should have been fixed, but apparently isn't. Or am I completely missing something? I'm using Ubuntu 9.04, rsync version 3.0.5, protocol version 30. Regards, Enno -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync with danish chars in filename
Tomas Norre Mikkelsen (tnm.p...@gmail.com) wrote on 17 September 2009 09:31: I have a Qnap NAS-219 used for storage, i have a Ubuntu 9.04 server for backup. When rsync between them filenames containing ø have problemes, the danish chars æ å does not have any problems eventhough its a special char like æ ø å. The log files state following: file has vanished: /M?deskabelon.doc where ? should be ø, and the file is stil at the system, so its not vanished. The charset on NAS is: en_US.UTF-8 And backup-server is: en_DK.UTF-8 I know there is a difference but in the starting wizard at the nas, it was the recommended charset used for danish language files. And the problem had been more understandable if the charset was like: The charset on NAS is: en_DK.UTF-8 And backup-server is: en_US.UTF-8 then the backup-server should not understand danish special chars like æ ø å. If you're only doing backups and don't intend to access the copies often, you could just ignore any charset/locale/i18n issue and use only plain C locale. No iconv anywhere, etc. Use the same settings at both sides of course, so that neither the sender nor the receiver tries to do any translation. I use --no-8-bit-output --no-iconv. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: DO NOT REPLY [Bug 6741] 'deleting' messages show up in improper places
samba-b...@samba.org (samba-b...@samba.org) wrote on 17 September 2009 09:24: The 'deleting' messages below are showing up in the middle of a transfer. It shows files being deleted and the progress bar appears to detail the deleting process. Usually rsync handles files in alphanumeric order, but there are still a number of files to transfer in the Dio directory. samba-b...@samba.org (samba-b...@samba.org) wrote on 17 September 2009 09:24: --- Comment #1 from asch...@gmail.com 2009-09-17 09:24 CST --- a...@eee1:~$ rsync --version rsync version 3.0.5 protocol version 30 * If both ends are running version 3 rsync will by default choose --delete-during (AKA --del), producing what you see. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: time variable throttling rsync traffic
On 9/16/2009 11:11 AM, Andrew Gideon wrote: On Tue, 15 Sep 2009 22:01:04 +, Andrew Gideon wrote: It can also potentially be extended in other directions. For one crazy example, the utility (or some other utility that modifies the first utilities configuration) could listen on a port for messages from - presumably - the receiving server. That would be a way for the receiving server to tell the sender what bandwidth to consume. Having given this a little more thought, I see issues. The largest is that it requires that the receiving device be able to connect to a port on the sending device. In general, I'd expect firewalls to be a problem. There's a work around that I think started with the gaming community. The sending device needs to send a message to the receiving device which causes the sending device's firewall(s) to create an opening through which the receiver can then send messages. All this is just a work around, avoiding the real solution: some way for the two intermediary processes - the one invoked via --rsh on the client and the one presumably invoked via --rsync-path on the server - to communicate over the same SSH stream used by the two rsync processes. Hmm...perhaps a middle ground is for the SSH process on the sending side to permit the receiving device to talk back to the sending side via use of SSH's -R port forwarding option? Lee Winter and I had a conversation last night by phone about this very topic. Turns out that a fairly small set of changes enables all whole bunch of control. For example, if you have a single file (one to an rsync process) in which you place the bandwidth limit of that rsync process and you sample that file on a regular basis for change, you can modulate the data rate for that given process by changing the contents of that file. The next level of control requires an independent demon which allocates bandwidth on a per process basis according to some set maximum. As rsync processes come or go, the bandwidth allocation would shift among the remaining processes. If you allow the control processes to talk to each other and perform speed tests you can allocate according to the link and/or limits on both sides. If you let an rsync process fill a second say this file with how much is left to be transferred, you could finish up almost on processes at a higher bandwidth allocation than those that will be running much longer. Yet, seems counterintuitive but it lets you complete short running tasks faster and let them get out of the way which may (or may not) provide benefits. by adding relatively simple signals, bandwidth can be modulated by various mechanisms of varying complexity ranging from the very simple to the very complex. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Proposed tweaks
Lee Winter (lee.j.i.win...@gmail.com) wrote on 16 September 2009 01:16: The use case that needs some optimization is that of online repositories -- mirrors. In contrast to other kinds of usage such as file synchronization, replication, backup, etc., mirrors present a quite different set of needs. Yes. And the solution, if you want to go to the many pains of optimizing this case, is to use a script to drive rsync. The issue with the current implementation of rsync is that it imposes a heavy load on the source mirror (sender in rsync terminology). The load is composed of two components, one being IO necessary to scan the file system and the other being the computational cost of the delta calculations. The computational cost is negligible, as you admit yourself (quoted below). The scan is caused by mirror admins lack of knowledge; it can be avoided completely if the client script and the upstream admin do their job well. This is an old problem that was already very well solved. It dates from the big ftp sites on the internet some 15-20 years ago. Look at the mirror script by Lee McLoughlin (and an important patch by Ian Maclaine-Cross). In Debian it's the mirror package. However rsync is so much better than ftp that everybody started to use it and forgot to keep the previous technology that should still be used, just in combination with rsync instead of ftp. That's what I do here (and only discovered the previous work afterwards...). 1. CPU performance is increasing faster than disk performance, so eliminating the IO burden is the bigger win. 2. Repositories tend to have files that are already fairly dense. So they probably don't benefit all that much from the delta handling. So if the basis file can't be swapped to the sender then the computational load can still be eliminated by using --whole-file mode despite the small loss in transport efficiency. I admit that I have not tested this premise. Easily noticeable. If you had you wouldn't have said it :-) :-) Lee Winter (lee.j.i.win...@gmail.com) wrote on 16 September 2009 10:13: On Wed, Sep 16, 2009 at 1:44 AM, Matt McCutchen m...@mattmccutchen.net wrote: Both of your proposals have been discussed before (see below), Good. but neither has been taken very far because they would both involve large changes to rsync. The delta computations are addressed below. Externalizing the file list should have quite minimal impact on the existing code. The big problem with this approach is not the file list, which can (and should) be easily generated separately by mirrors, as I said above. The difficulty is that it makes the update process transactional; to have reliability you have to deal with all sorts of failures. The vast majority of mirrors use a ~10-line script, which is easy to write and maintain. Ours is so much more efficient and (I believe) as reliable as plain rsync, but is an 800-line monster full of subtleties, where a seemingly innocent change may corrupt your mirror and may only be noticed months later. How many admins would use it? Forget it... That's why rsync doesn't do it, even though there have been demands for a long time. This is a good idea that has been discussed before and is implemented in another tool called zsync. Wayne noted some of its drawbacks here: http://markmail.org/message/pt354zo4njgmupj Perhaps you could respond to his concerns. Second, I looked at the comments, and while technically valid I don't think they are reasonable. They're really excellent and reflect very well what happens in practice. - quote start (2) This increases the disk I/O on the sending side because it would need to read the file twice (assuming that cached checksums aren't available). Since rsync is running in a pipelined mode, the sender will be iterating over several future files by the time it gets back a request for data chucks from the receiving side. Hopefully the data will still be in the disk cache, but if it is not, the transfer would bog down to a significant degree. - quote stop Again we have a non-quantitative statement. The absolute worst case effect is to double the disk IO. I have a hard time imagining an rsync system that is within a factor of two of being disk-bound. But that's exactly what happens. quoting out of order I was actually proposing a more ambitious change which would make the directionality the subject of an user option. That seriously increases the scope of the change, but it also gains you the benefit of the feature (flexible direction), so it might be justified on that basis. The matter is so important that I'd block it at the server side, even if I had to patch the program. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync-3.0.6 on cygwin-1.7
A new 3.0.6 package is now available on cygwin-1.7; I don't plan to upgrade Cygwin-1.5 packages anymore, so if you want newer releases on rsync please upgrade to 1.7, with proper care: http://cygwin.com/#beta-test This release includes a small patch on rsync configure to detect IPv6 and xattr on Cygwin, please consider including them (or a modification of them) upstream to help people have a full-featured rsync on cygwin also when compiling it manually and not thru the cygport package. xattr right now should work on SMB and will be fixed on NTFS very soon. The xattrs test fails because Cygwin doesn't currently have a setfattr utility, so right now the xattr feature is not thoroughly tested and is considered experimental. http://thread.gmane.org/gmane.os.cygwin/110581 --- origsrc/rsync-3.0.6/configure.in2009-05-08 19:07:14.0 +0200 +++ src/rsync-3.0.6/configure.in2009-09-17 06:53:08.927125000 +0200 @@ -196,8 +196,18 @@ AC_ARG_ENABLE(ipv6, [don't even try to use IPv6])) if test x$enable_ipv6 != xno; then AC_MSG_CHECKING([ipv6 stack type]) - for i in inria kame linux-glibc linux-inet6 toshiba v6d zeta; do + for i in cygwin inria kame linux-glibc linux-inet6 toshiba v6d zeta; do case $i in + cygwin) + AC_EGREP_CPP(yes, [ +#include netinet/in.h +#ifdef _CYGWIN_IN6_H +yes +#endif], + [ipv6type=$i; + AC_DEFINE(INET6, 1, [true if you have IPv6]) + ]) + ;; inria) # http://www.kame.net/ AC_EGREP_CPP(yes, [ @@ -940,7 +950,7 @@ if test x$enable_xattr_support = xno AC_MSG_RESULT(no) else case $host_os in -*linux*) +*linux* | *cygwin*) AC_MSG_RESULT(Using Linux xattrs) AC_DEFINE(HAVE_LINUX_XATTRS, 1, [True if you have Linux xattrs]) AC_DEFINE(SUPPORT_XATTRS, 1) -- Lapo Luchini - http://lapo.it/ signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
