Re: retransfer fail of large files with inplace and broken pipe
On Sat, Dec 12, 2009 at 11:21 PM, tom raschel rasc...@edvantice.de wrote: so i had a look at --inplace which I thougt could do the trick, but inplace is updating the timestamp and if the script start a retransfer after a broken pipe it fails because the --inplace file is newer than the original file of the sender. Are you using --update (-u)? If so, turn that off. If not, rsync won't skip a file that is newer, so something else is afoot. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: multiplexing overflow errors
On Fri, Dec 4, 2009 at 11:26 PM, Matt McCutchen m...@mattmccutchen.netwrote: Here is the FAQ patch. Thanks! I've updated the FAQ. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: A crash when syncing xattr with rsync-3.0.6
On Mon, Nov 30, 2009 at 4:46 AM, Amir Rapson amir.rap...@gmail.com wrote: I have a crash running rsync-3.0.6 and using the -X flag. This has been fixed in git since mid August (with the freeing of the proper pointer, not the removal of the free() call.) See the b3.0.x branch. I'm thinking about releasing a 3.0.7 with that fix and a few others. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Don't (wrongly) retouch dir permissions with --fake-super.
On Sun, Nov 29, 2009 at 5:26 PM, Chris Adams cmad...@hiwaay.net wrote: Hmm, I've applied this patch on both sides, but I'm still getting the same error. The errors should go away on the next run, since rsync should have set u+rwx on the dirs during the most recent run (but it doesn't do it soon enough to avoid xattr read errors). I noticed that this could happen in my testing, but didn't see any particular need to fix it because the fixed rsync won't create dirs that way in the first place, so it should be a transitional error. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Don't (wrongly) retouch dir permissions with --fake-super.
On Sat, Nov 28, 2009 at 2:17 PM, Matt McCutchen m...@mattmccutchen.netwrote: diff --git a/generator.c b/generator.c Much appreciated. Applied and tested. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: multiplexing overflow errors
On Fri, Nov 27, 2009 at 7:45 AM, Dieter Stüken d.stue...@conterra.dewrote: log/20091123_223901/dresden.log:multiplexing overflow 101:7104843 [sender] This has come up before. That number translates into the byte sequence 0x4b 0x69 0x6c 0x6c, which is Kill in ASCII. So, something is outputting a Killed ... message to stdout instead of the proper stderr (which you may want to report to the Solaris folks so they can fix it). To fix the rsync-copy failures, you should figure out why rsync is being killed. Perhaps your system logs will tell you. Or maybe attaching a syscall-tracing utility to the appropriate process would reveal the whole message. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Copiying /etc/shadow with sudo and rsync
On Tue, Nov 24, 2009 at 1:49 AM, Gerard Mensoif mensoif_ger...@yahoo.frwrote: rsync --rsync-path='sudo rsync' --rsh='/usr/bin/ssh -x' --archive u...@localhost:/etc/shadow u...@server2:/etc/shadow One end of the transfer has to be local (even a host named localhost is not local), so one thing you can do is to use two rsync commands (pull, then push). Or just make sure that the sending side of the transfer is running rsync as root (e.g. use sudo to run the local rsync too). Also, a modern rsync would have exited with an error about your invalid two-server syntax, so you may wish to upgrade to something newer. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exit status 13 in version 3.1
On Thu, Nov 12, 2009 at 10:00 PM, Matt McCutchen m...@mattmccutchen.netwrote: My point is that it's a clunky way to achieve the goal, and it would be simpler for the sender to just keep reading after a write error. Yeah, that's a good idiom, and the latest code wasn't doing a good enough job of that when the socket closed. One new issue turned out to be that the client started getting a read error (where the old code would get a write error) due to the buffer-filling code getting an EOF on the socket (even though the perform_io() call was ostensibly there to try to write some data). There was also a bug where an EOF error did not try to empty out any message data currenly in the buffer. I've committed some improvements for this, so the error reporting should be better when talking to older versions. For instance, the input code now just notes an input EOF unless it is there to read some data and the data is not present. It also does a better job of draining any pending error messages from the input buffer (and reading any final messages) when a write error occurs. As for the arbitrary io_timeout that gets set, yes, it is debatable if it is a good idea. While not strictly necessary, I kind of like having it just as a final sanity check for the error-exit code. I did make the arbitrary timeout larger, though, since it should not ever really be needed. I'll also review the error-handling a bit more, and possibly reconsider this further. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Daemon tests broken when running as root
On Thu, Nov 12, 2009 at 9:11 PM, Matt McCutchen m...@mattmccutchen.netwrote: I'm guessing the uid = 0 and gid = 0 in the test daemon configuration were disabled so that the daemon tests could run for unprivileged users. Right. I've checked in a fix that ensures that uid gid are specified when the test is run as root. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Does files-from work with --delete?
On Thu, Nov 12, 2009 at 6:07 PM, Matt McCutchen m...@mattmccutchen.netwrote: The best approach for now is probably to backport the --delete-missing-args changes to 3.0.6. In the future, I'd suggest starting with the head of the b3.0.x branch. That currently gets you one extra commit, an xattr-related memory fix (2daed024b17a2cafb956e12581c25119d07a5950). ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Does files-from work with --delete?
On Thu, Nov 12, 2009 at 3:56 PM, Philip Pokorny ppoko...@penguincomputing.com wrote: How confident are you in the current state of 3.1.0.pre and the nightly snapshots? Should I be concerned about running this on production data? Personally, I'm almost ready to start using it in production. The 3.1.0dev code prior to the I/O changes was in production-ready shape, and the I/O overhaul has been testing well so far (I use it for all the personal rsyncing needs). Any failure cases from the I/O code should result in the stopping of the transfer, not some kind of corruption, and I haven't found anything in quite a while. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exit status 13 in version 3.1
On Thu, Nov 12, 2009 at 5:47 PM, Matt McCutchen m...@mattmccutchen.netwrote: It looks like the implementation has the receiver hang around for a hard-coded 10 seconds, accepting data from the sender and discarding it. No, it sets a timeout of 10 seconds (i.e. 10 seconds of inactivity), which in the new protocol should never be reached because the we're exiting with an error message gets everyone to die in unison. The necessity of discarding data is there due to the pipelining nature of rsync, particularly if the error is coming from the receiver. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another include/exclude question
On Sun, Nov 8, 2009 at 10:21 PM, Thomas Gutzler thomas.gutz...@gmail.comwrote: I thought --include=/this_dir/ --include=/this_dir/*** would do it, but it doesn't. The exclude * seems to overwrite the include matches: [sender] hiding file this_dir/foo because of pattern * Order is important. Whatever matches first, is what takes effect. Also, *** matches both the dir and its contents, so you could use: --include='/this_dir/***' --exclude=* As long as this_dir is in the root of the transfer, that will work. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exit status 13 in version 3.1
On Wed, Nov 4, 2009 at 8:05 PM, Matt McCutchen m...@mattmccutchen.netwrote: With commit 84c11e85a4c4a12ecacba24afe9617222e4361e6, I get different output, but still not the desired No space left on device: Yeah, that's the long-standing issue where a fatal error on the server side can cause the client side to get a socket error trying to write to the socket before it has a chance to read the error(s) from the socket. The latest git archive finally has a fix for this. It should make the exit messages much cleaner for an abnormally-exiting protocol 31 transfer, both receiving the final error from the server and exiting with the proper error code. When an older client is talking to a new server, the error should also make it to the client, but the client will exit with the (sadly) traditional error about an unexpectedly closed connection. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exit status 13 in version 3.1
On Tue, Nov 3, 2009 at 9:25 AM, Matt McCutchen m...@mattmccutchen.netwrote: rsync error: errors with program diagnostics (code 13) at log.c(340) [sender=3.1.0dev] This means that you didn't update recently. Sadly, it appears my reply mentioning that I fixed the problem only went to Carlos, and missed the list. Here it is: Wayne said: That is likely to have been caused by the code going into --msgs2stderr mode when it gets a socket error, but rwrite() couldn't handle some of the more esoteric error codes (because they don't normally make it that far without being transformed into a simpler code). I've checked in a fix for this. If you would, please re-check your disk-overflow test case with the latest code (and --msgs2stderr turned off) and see if that gets the error to you. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: strange log values in 3.1
On Sun, Oct 25, 2009 at 6:26 PM, Carlos Carvalho car...@fisica.ufpr.brwrote: All files are new. What I find strange is that I'd expect the transfered bytes to be somewhat larger than the file sizes Yeah, there are two problems. One was that the %b modifier requires late-logging (after the transfer finishes), and the log-format scanner hadn't been updated to understand apostrophes, so it wouldn't find %'b. If nothing else triggered the after-the transfer output, you would have seen some constant value (like 0) for every file. The second issue is the reason you saw several files at 0 followed by some too-big numbers: the %b and %c routines are outputting differences in stats read/written values, and the newest I/O code counts these bytes as they are read/written from/to the socket, so the buffering gets in the way of the per-files stats. I've checked in a fix for the apostrophe issue, and am working up a fix for the per-transfer stats. Thanks for the report! ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: bugs in version 3.1
On Fri, Oct 23, 2009 at 2:41 PM, Carlos Carvalho car...@fisica.ufpr.brwrote: The first is that when running with --files-from and -ii unmodified files are not put in the log. That is not a logging issue, but rather an early-termination issue due to a protocol problem when speaking protocol 30. rsync: io.c:990: start_filesfrom_forwarding: Assertion `(iobuf.out_empty_len != 0)' failed. That bug showed that the --files-from data-shipping code needed to handle a non-multiplexed socket for protocol 29. I did some work on backward-compatibility issues between --files-from and the new I/O code, and I fixed quite a few bugs (both new and old), including the two above. I think that --files-from should be in good shape now. Some of my recently fixes were in the area of iconv (including iconv translations of files-from data). There were a few things I discovered that weren't handled right in all prior versions of rsync that the latest 3.1.0dev will now handle correctly (such as properly translating a multi-byte character that spans the end of a read buffer). ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: PATCH: --write-devices to allow synchronising to a block device
On Tue, Oct 20, 2009 at 8:28 PM, Darryl Dixon - Winterhouse Consulting darryl.di...@winterhouseconsulting.com wrote: I haven't had any response to this mail over the last week - does no-one have any comments or opinion? I highly doubt my code is pristine and perfect ;) Is there a possibility that this patch could be included in the general rsync-patches distribution? Hey there! Thanks for the patch. I haven't had a chance to try it, but I did snag it, clean it up, and make it apply to the current git source. Note: it usually helps to use an attachment for a patch so that it doesn't get manged; i.e. your patch was wrapped and had all tabs replaced by spaces. You'll find the patch in the rsync patches repo: http://gitweb.samba.org/?p=rsync-patches.git;a=tree ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Enhanced authentication and authorization in rsyncd
So, I've tweaked your patch, and you'll find the results in the git version of rsync and (currently) in a patch. Here are some details: 1. allow the uid and gid used to access a certain module to be determined by the authenticated user - uid = __auth__ will use the auth_user's uid gid = __auth__ will use the auth user's main group's gid I changed this to use a new environment-variable expanding idiom in the config file. So, the above would be written like this: uid = %RSYNC_USER_NAME% gid = * The * setting was an already-implemented feature in the git repo which expands into the uid user's groups. 2. seteuid and not just setuid Applied. 3. added rw users to allow read/write access to a module, ro users to allow read only access to a module and deny users. read only on the module overrides the user's authorization. 4. added support for groups - with a '@' prefix. For instance: auth users = tridge, susan, @rsync_users I changed this to keep a single auth users setting, but did not commit it to git (at least, not yet). I created a patch named group-auth.diff, which you can find here: http://gitweb.samba.org/?p=rsync-patches.git;a=tree These patches apply to the current git-repo version of rsync. In the group-auth patch you'll find your @group idiom for specifying a group name (as in your patch, if the username is a real user, that real-user's groups are compared against the @group items). I changed your ro, rw, and deny settings to be specified via suffixes. For example: auth user = joe:deny admin:rw @rsync:ro susan This makes the logic very easy to figure out, since we just match each item in order and go with the first match. That makes the deny suffix's job to override an authorization that would succeed in a later match. In my patch, the module's read only setting is the default, and the ro/rw suffix of the first matching rule overrides that value. I also added the ability to put a @group password in the secrets file, so if you define @rsync:secret, that lets anyone in the rsync group login with that password (a group-matching rule looks for either a user password or a group password, while a user-matching rule only looks for a user password). Potential backward incompatibility: usernames used to be able to start with a @ and (surprisingly, given the syntax of the secrets file) even contain a colon. If anyone likes/dislikes this, please feel free to comment. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Are there any current, in play plans, to multi-thread rsync?
On Fri, Oct 16, 2009 at 8:32 PM, Michael Brian - IL micha...@tusc.comwrote: I am looking for an enhancement to multi-thread or parallelize rsync. Are there any plans that would allow one to set a parameter, like 'number_of_threads' and rsync will ship multiple files at the same time? Not at present. That would be a pretty big protocol change, and is probably something that would be best left to a new-protocol rewrite. It would be quite interesting to have separate connections for the hierarchy traversal code versus one or more file transfer connections that were controlled by the traversal process, but that is not something that I'm working on, nor have I heard about anyone else doing that. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: retrieve files without local dir compare
On Wed, Oct 14, 2009 at 2:36 AM, Marc Mertes mer...@uni-bonn.de wrote: If the data is here, it is imported by a special software, after the import it will be deleted from that directory. The deleting can't be disabled. If you know what file just got processed, append its name onto an exclude file (one per line), and then use --exclude-from=EXCLUDE_FILE. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Disable checksumming to improve local performance?
On Fri, Sep 04, 2009 at 10:26:43AM -0700, Greg Siekas wrote:
Is it possible to disable checksumming? I'm using rsync with -W,
whole file, so if a file has changed I don't want to just transfer
the changes.
There is not currently a way to do that. I whipped up the appended
patch that makes -WW disable full-file checksum computation and makes
that the default for a local transfer. When comparing -W vs -WW in some
local-transfer testing, it didn't result in any perceivable difference
in transfer speed or runtime. You can verify that it is enabled by
using --out-format='%i %C %n%L' and noting that all the checksums turn
into all-bits-on values (the checksum bytes are still transmitted so
that a read-error on the sending side can be indicated to the receiver).
..wayne..
index 6655acd..d8b228d 100644
--- a/compat.c
+++ b/compat.c
@@ -34,6 +34,7 @@ extern int use_qsort;
extern int allow_inc_recurse;
extern int append_mode;
extern int fuzzy_basis;
+extern int whole_file;
extern int read_batch;
extern int delay_updates;
extern int checksum_seed;
@@ -286,6 +287,8 @@ void setup_protocol(int f_out,int f_in)
receiver_symlink_times = 1;
#endif
}
+ if (whole_file 1 protocol_version 31)
+ whole_file = 1;
if (need_unsorted_flist (!am_sender || inc_recurse))
unsort_ndx = ++file_extra_cnt;
index 407568d..72636eb 100644
--- a/main.c
+++ b/main.c
@@ -519,7 +519,7 @@ static pid_t do_cmd(char *cmd, char *machine, char *user, char **remote_argv, in
/* If the user didn't request --[no-]whole-file, force
* it on, but only if we're not batch processing. */
if (whole_file 0 !write_batch)
- whole_file = 1;
+ whole_file = 2;
set_allow_inc_recurse();
pid = local_child(argc, args, f_in_p, f_out_p, child_main);
#ifdef ICONV_CONST
index 611035f..2c356e3 100644
--- a/match.c
+++ b/match.c
@@ -25,6 +25,7 @@
extern int checksum_seed;
extern int append_mode;
extern int checksum_len;
+extern int whole_file;
int updating_basis_file;
char sender_file_sum[MAX_DIGEST_LEN];
@@ -124,9 +125,11 @@ static void matched(int f, struct sum_struct *s, struct map_struct *buf,
n += s-sums[i].len;
}
- for (j = 0; j n; j += CHUNK_SIZE) {
- int32 n1 = MIN(CHUNK_SIZE, n - j);
- sum_update(map_ptr(buf, last_match + j, n1), n1);
+ if (whole_file 2) {
+ for (j = 0; j n; j += CHUNK_SIZE) {
+ int32 n1 = MIN(CHUNK_SIZE, n - j);
+ sum_update(map_ptr(buf, last_match + j, n1), n1);
+ }
}
if (i = 0)
@@ -336,7 +339,10 @@ void match_sums(int f, struct sum_struct *s, struct map_struct *buf, OFF_T len)
matches = 0;
data_transfer = 0;
- sum_init(checksum_seed);
+ if (whole_file 2)
+ sum_init(checksum_seed);
+ else
+ memset(sender_file_sum, 0xFF, checksum_len);
if (append_mode 0) {
if (append_mode == 2) {
@@ -377,7 +383,7 @@ void match_sums(int f, struct sum_struct *s, struct map_struct *buf, OFF_T len)
matched(f, s, buf, len, -1);
}
- if (sum_end(sender_file_sum) != checksum_len)
+ if (whole_file 2 sum_end(sender_file_sum) != checksum_len)
overflow_exit(checksum_len); /* Impossible... */
/* If we had a read error, send a bad checksum. We use all bits
index 66820b5..f9b1939 100644
--- a/options.c
+++ b/options.c
@@ -34,14 +34,9 @@ extern filter_rule_list daemon_filter_list;
int make_backups = 0;
-/**
- * If 1, send the whole file as literal data rather than trying to
- * create an incremental diff.
- *
- * If -1, then look at whether we're local or remote and go by that.
- *
- * @sa disable_deltas_p()
- **/
+/* If 1, send the whole file as literal data rather than trying to
+ * create an incremental diff. If 1, disable full-file checksumming.
+ * If -1, then look at whether we're local or remote and go by that. */
int whole_file = -1;
int append_mode = 0;
@@ -928,7 +923,7 @@ static struct poptOption long_options[] = {
{exclude-from, 0, POPT_ARG_STRING, 0, OPT_EXCLUDE_FROM, 0, 0 },
{include-from, 0, POPT_ARG_STRING, 0, OPT_INCLUDE_FROM, 0, 0 },
{cvs-exclude, 'C', POPT_ARG_NONE, cvs_exclude, 0, 0, 0 },
- {whole-file, 'W', POPT_ARG_VAL,whole_file, 1, 0, 0 },
+ {whole-file, 'W', POPT_ARG_NONE, 0, 'W', 0, 0 },
{no-whole-file,0, POPT_ARG_VAL,whole_file, 0, 0, 0 },
{no-W, 0, POPT_ARG_VAL,whole_file, 0, 0, 0 },
{checksum,'c', POPT_ARG_VAL,always_checksum, 1, 0, 0 },
@@ -1499,6 +1494,12 @@ int parse_arguments(int *argc_p, const char ***argv_p)
}
break;
+ case 'W':
+ if (whole_file 0)
+whole_file = 0;
+ whole_file++;
+ break;
+
case 'P':
if (refused_partial || refused_progress) {
create_refuse_error(refused_partial
@@ -2288,11 +2289,14 @@ void server_options(char **args, int *argc_p)
argstr[x++] = 'k';
}
- if (whole_file 0)
- argstr[x++] = 'W';
/* We don't need to send --no-whole-file, because it's the
* default for remote transfers, and in any case old versions
* of rsync will not understand it. */
+ if (whole_file 0)
Re: Enhanced authentication and authorization in rsyncd
On Sun, Aug 30, 2009 at 12:06:21PM +0300, Amir Rapson wrote: A slightly better patch file (removed some warnings). Thanks! Sorry for the slow reply, but your patch looks very useful. I'm reviewing it for inclusion in 3.1.0. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem with 3.0.6 and fileflags
On Wed, Jul 29, 2009 at 03:24:57PM -0400, Roger Bailey wrote: I am having trouble compiling rsync 3.0.6 on Red hat Linux EL 5.3. [...] I am puzzled by the last entry, no file-flags. That file-flags support is for BSD, so configure determines that it isn't available for Linux. Linux has something similar (controlled by chattr), but support for that hasn't been integrated into the patch. I took a look at it once, and ran into some complications, and didn't finish it. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How could be total transferred file size more than total file size ?
On Wed, Jul 29, 2009 at 11:17:34AM +0530, paresh masani wrote: Could you please tell me how would it be possible that total transferred file size is more than total file size ? I believe that the only way this can happen is for a resend to occur (when rsync notices that a file doesn't match the checksum and transfers it again). ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: (Recent I/O change)
On Mon, Oct 5, 2009 at 1:51 AM, Matt McCutchen m...@mattmccutchen.netwrote: Would you care to explain the impact of this change on the rsync I/O design in terms that I might understand? It looks like you merged two switch statements over possible message types, reducing code duplication, but that's all I can really tell. In addition to simplifying the code, it makes the message pipe more robust. The old code just sent raw file-list data (of unspecified length) as an implied part of the MSG_FLIST message. The new code packs raw data into MSG_DATA messages, which will allow other messages to intermingle with the data. If the code had been working like this from the start of 3.0, I could have fixed the issue where an I/O error isn't getting noticed soon enough just by having the sender send its I/O error message before the end of the file-list data. As for the unsightly files-from code block, that will be changing soon, since I'm currently working on simplifying the I/O code. The new code will mainly use a single buffered-I/O function for all the normal pipelining of data, making it simpler, less likely to jam up, and hopefully more performant too. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Old versions of lib/addrinfo.h missing from rsync repo
On Wed, Sep 02, 2009 at 12:32:45AM -0400, Matt McCutchen wrote: Why were they dropped from the development history? There doesn't appear to be a copyright issue: the file bears a BSD license. It used a 3-clause BSD license, which is incompatible with the GPL. The incompatible code was expunged from CVS and was never imported into git. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: content of file
On Sun, Aug 30, 2009 at 02:53:12PM -0400, Mag Gam wrote: Instead of copying the whole file, I rather use some sort of pipe. thats all I assume you're using an rsync daemon, otherwise you could just use ssh host tail -f file or similar. If data is always being appended, see the --append option for a modern rsync (e.g. at least 3.x) which will just grab the most recent data and tack it onto the end of the local copy of the file. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsyncd always compression
On Mon, Aug 31, 2009 at 12:55:56PM +0200, Uwe Driessen wrote: Anyone knows a trick that the server only answer if the client is use the compression? This is not currently possible. It seems like it would be nice to have a require options = --compress setting that is the opposite of the refuse options setting. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Include a directory inside an excluded one?
On Thu, Aug 20, 2009 at 08:41:40PM +0200, Nicolas KOWALSKI wrote: My problem is that I would like to synchronize a directory located inside /Ma musique/, the one named /Ma musique/iTunes/. As you noticed, rsync will not synchronize anything whose parent dir gets excluded. To fix this, stop excluding the parent dir and instead exclude the content of the dir (but after any includes from the dir): --include=/Ma musique/iTunes/ --exclude=/Ma musique/* ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --xattrs requires protocol 30 or higher error
On Sat, Aug 22, 2009 at 11:22:40AM +0100, Kirk Maule wrote: Both rsync versions are 3.0.6 protocol version 30. You'll undoubtedly find that there is an older rsync on the $PATH on one of the machines. e.g, try this: ssh hostname rsync --version Either get rid of the old version, or specify --rsync-path=/path/rsync so that you get the rsync you expect. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Differenc in exclude v 2.6.9 and v 3.0.4
On Thu, Jul 09, 2009 at 12:42:47AM -0500, Harry Putnam wrote: Is this actually a difference that was one way in an earlier (2.6.9) version and is different in the later (3.0.4) version? That seems unlikely to me... Yes, very unlikely. The combination of the options --include=Temp/**/ and --exclude=Temp/** works fine for me. You used -vv, so you should see any excluded files/directories mentioned. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: partial-dir not being used?!
On Fri, Jul 17, 2009 at 02:13:00AM -0300, Carlos Carvalho wrote: There can be several partials in the partial dir for the same file, resulting from more than one interruption. No, there is only one partial file. Don't confuse a partial file with a temp file, which are different things in rsync's nomenclature. A temp file is where rsync constructs the new destination file, and if it gets interrupted (and --partial is set) it either saves the temp file over the destination file, or (with --partial-dir), save the temporary file into the partial dir with the same name as the destination file. It is this moving of the temp file that turns it into a partial file that rsync can use in the next transfer. The use of --delete will clean up any left-over temp files if an rsync got forcibly killed (so it couldn't clean up the temp file). What happens if we set temp-dir the same as partial-dir? Rsync does not currently support a relative temp-dir if it does not exist in each directory. It would be possible to change this so that rsync goes around creating a relative temp dir for every dir that needs a transfer (and removing it afterwards), but the current code doesn't do that. (It only creates a missing dir if the transfer gets interrupted and it needs to save a partial file into the --partial-dir.) ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --xattrs requires protocol 30 or higher error
On Sat, Aug 22, 2009 at 08:49:10PM +0100, Kirk Maule wrote: Should the rsync*.tar.gz install procedure replaced rsyncd 2.6.8 as well as rsync 2.6.8 with the 3.0.6 version? Depends on how the package was built. Rsync is just one executable, so be sure to put it in a consistent spot. For your daemon setup, check your xinetd setup and/or init file to see what they are running. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsyncd.conf chroot yes problem with symlink-ing
On Fri, Aug 14, 2009 at 01:03:35PM +0300, alex.vladule...@csteam.ro wrote: ls -l /home/webmaster/domains/csteam.ro lrwxrwxrwx 1 root root 31 2009-07-28 21:47 /home/webmaster/domains/csteam.ro - /home/csteam/domains/csteam.ro/ Right. That can't work unless you set use chroot = false since the symlink is not a link that works inside the chrooted area. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete flag not working on cygwin rsync
On Tue, Aug 04, 2009 at 03:10:20PM -0400, Sharad K wrote: Everything works fine except the --delete flag. Does rsync output a warning about an I/O error, saying it is disabling deletes? You can use --force if you're sure you want rsync to override that. You might also try using --dry-run to see if rsync thinks it will remove a file, but doesn't actually do so in the real run (newer rsync versions are more verbose about skipped deletions). ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync speedup - how ?
On Thu, Aug 06, 2009 at 08:15:39PM +0200, devz...@web.de wrote: i read that rsync would be not very efficient with ultra-large files (i`m syncing files with up to 80gb size) Things to try: - Be sure you're using rsync 3.x, as it has a better hash algorithm for the large numbers of checksum blocks that need be scanned on the sending side. - The --inplace option might help, since it can reduce the amount of write I/O when the file is being modified (though it does reduce the amount of backward matching). In a really large file where most of the data stays the same, this could be a big win. - Try setting the --block-size option. This will only help if the block size is so large it is missing finding matching data. In a huge file that is mostly unchanged, this may not be an issue. Note that decreasing the block size increases the amount of checksum data, and the amount of blocks in the matching algorithm. - The best things you could do would be to mount the virtual drives (source read-only, dest read/write) and copy within the file systems. That would allow rsync to use its size+mtime fast-check to skip most of the files. It would not, however, result in truly identical disk images, so may not be a solution for you. Keep in mind that the checksumming as it currently works requires the receiving side to read the whole file (sending its checksums), then (after that is done) the sending side reads the whole file (generating differences), which allows the receiving side to reconstruct the file while the sender is sending in the changes. Sadly, this means that the transfer serializes this file-reading time (since the sender wants to be able to find moved blocks from anywhere in the file). An interesting new option might be one that tells the sender to immediately start comparing the received checksums to the source file, and only check if the data matches (with no movement) or if it needs to send the changed data (i.e. this would skip scanning for moved data). For mostly unchanged, large files, that would allow concurrent reading of the receiving and sending files. Combined with --inplace, this might be a pretty large speedup for mostly-unchanged files. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: supporting --fake-super on opensolaris (zfs) destination
On Mon, Jul 27, 2009 at 07:30:48AM +, Enrico Cavalli wrote: I wrote a small patch in order to support what I think is an absolutely needed feature in order to make rsync-based backups retaining complete ownership permission when writing to an opensolaris machine using --fake-super. Thanks for the preliminary Solaris xattr-support code. I've cleaned it up a bit, and will be checking it into the git repository soon. Let us know how things progress towards full -X support. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: What happened to librsync?
On Tue, Jul 14, 2009 at 11:49:32PM -0600, Sam Howard wrote: It seems that librsync got left by the roadside ... is there anything similar for rsync 3.x.x? The librsync code doesn't implement the rsync protocol, but just some differential transfer routines similar to rsync's basic algorithm. It can be used to make an incompatible copy program, such as my test code that I called rZync (that code works for what it does, but it isn't a finished program by any means). ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: are partial files always deleted on interruption?
On Fri, Jul 03, 2009 at 03:32:29PM -0300, Carlos Carvalho wrote: Both processes were killed with the same command, kill pid1 pid2. A SIGTERM (the default kill signal) should be caught by rsync, allowing it to cleanup before it exits. However I've found that a file was left after interruption, with a zero size. What kind of file? A temp file, or the destination file? You didn't use --inplace, right? Rsync 2.6.8 had a bug where it could leave a temp file when interrupted, but I don't see any such problem with 3.0.6. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: what happens with .~tmp~ after an interruption?
On Fri, Jul 03, 2009 at 05:25:18PM -0300, Carlos Carvalho wrote: What happens if [some of] these files no longer exist in the source when the next run happens? Are they, and .~tmp~, removed or kept? The file is only removed if a matching filename is transfered in the parent directory. This is because rsync doesn't know if the file was left by some other rsync that is going to still need it, or if it should clean it up. It might be nice to have rsync remove any files with a ctime older than a certain span of time, but that is not currently done. If your rsync runs are leaving .~tmp~ dirs lying around, you can either schedule your own aging of files in .~tmp~ dirs (e.g. via find), or use some custom rsync command that will only remove a .~tmp~ dir and its contents from a remote directory (e.g. via include/exclude rules). An example find command: find . -path '*/.~tmp~/*' -cmin 240 -delete That's leaves rsync to cleanup the empty dirs. You could use a more complex command to clean those too: find . -depth -path '*/.~tmp~/*' -cmin 240 -delete -o -name '.~tmp~' -cmin 240 -delete ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Bug? backup_dir sometimes reported on STDOUT, sometimes not.
On Fri, Jul 03, 2009 at 10:46:13PM +0200, Axel Essbaum wrote:
In the second case the backup_dir report in the output is apparently
not on STDOUT.
In all versions of rsync prior to 3.1.0 (which is not yet released), the
backup_dir is ... message is output very early in the transfer, so a
remote rsync cannot send it over the protocol (since it is not yet setup
right for messages). It instead sends it via STDERR (since STDOUT is
being used to talk to the client rsync). 3.1.0 fixes this by moving
this output later in the sequence of events, allowing the message to
come through the protocol and go to STDOUT as expected.
You have several options:
- Don't use double verbosity, and the message won't be output at all.
If you want to see unchanged files, you can try -ii as an alternative
way to get that info (though it does show even attribute changes, and
you'd need to be able to parse the new output format).
- Use 3.1.0 -- it's in a stable state, and I'm hoping to get it
released before too long. (I use it for all my rsyncing.)
- Patch the remote rsync to move that output from options.c to main.c.
(See the attached patch.)
..wayne..
--- a/main.c
+++ b/main.c
@@ -76,7 +76,9 @@ extern char *rsync_path;
extern char *shell_cmd;
extern char *batch_name;
extern char *password_file;
+extern char *backup_dir;
extern char curr_dir[MAXPATHLEN];
+extern char backup_dir_buf[MAXPATHLEN];
extern struct file_list *first_flist;
extern struct filter_list_struct daemon_filter_list;
@@ -752,6 +754,9 @@ static int do_recv(int f_in, int f_out, char *local_name)
exit_cleanup(RERR_IPC);
}
+ if (backup_dir verbose 1)
+ rprintf(FINFO, backup_dir is %s\n, backup_dir_buf);
+
io_flush(NORMAL_FLUSH);
if ((pid = do_fork()) == -1) {
--- a/options.c
+++ b/options.c
@@ -1504,8 +1504,6 @@ int parse_arguments(int *argc_p, const char ***argv_p)
backup_dir_buf[backup_dir_len++] = '/';
backup_dir_buf[backup_dir_len] = '\0';
}
- if (verbose 1 !am_sender)
- rprintf(FINFO, backup_dir is %s\n, backup_dir_buf);
} else if (!backup_suffix_len (!am_server || !am_sender)) {
snprintf(err_buf, sizeof err_buf,
--suffix cannot be a null string without --backup-dir\n);
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync flags freebsd
On Tue, Jun 30, 2009 at 09:42:08PM -0400, alexus wrote: I dont know if that helpful, but all I'm saying is fileflags.diff isn't working for rsync-3.0.6, but it works for rsync-3.0.5 It doesn't look like you download rsync-patches-3.0.6.tar.gz -- your patch command would have been patch -p1 patches/fileflags.diff if you had been using that version. Unpack both tar files in the same parent dir, since they both start with the rsync-3.0.6 top dir in the archive. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: difference between --no-implied-dir abd --keep-dirlinks ?
On Mon, Jun 29, 2009 at 10:33:34AM +0530, Jignesh Shah wrote: I have gone through the rsync documentation and also tested same both --no-implied-dir and --keep-dirlinks options. My question is what is the difference between these options in below scenario? Implied directories are those that are above the transfer area but included as a part of the path due to the use of -R (--relative). By default all implied directories get sent with their directory attributes, and the destination is forced to match that. If you want the implied paths to differ, --no-implied-dir will leave them alone. Using --keep-dirlinks also affects the transfer area, so that you can put a symlink to a directory anyplace that the sender has a dir. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [rsync] include-from with different source points
On Mon, Jun 29, 2009 at 09:43:02AM -0400, GravyFace wrote: sudo rsync --progress -avz --include-from=/home/gravyface/backup.list / /mnt/usbbackup/localbackup/ If you drop the '*'s from the lines in the file and switch to using --files-from, that should do what you were expecting. You still need to specify the / source because files-from files are all relative to the source. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which function in generator trigger the file transfer?
On Tue, Jun 02, 2009 at 03:31:02PM +0800, Daniel.Li wrote: As receiver will use read_ndx_and_attrs to read iflags to judge if it's need to be transfered? You can get a good overview of how rsync works here: http://rsync.samba.org/how-rsync-works.html After reading through that, you should discover that it's the generator that decides what work gets done. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --remove-source-files
On Tue, Jun 02, 2009 at 04:08:26PM +, C?sar Tejeda Hern?ndez wrote:
rsync -tai --remove-source-files --ignore-existing --no-g --no-o --no-p
u...@source:operativo/almacen/{ESP05/recorte/*.tar,EU/recorte/*.tar}
/media/BACKUP3_PROMES10/PROMES10km/
[...]
¿Is there any way to specify that rsync should not report an error if it
finds no files to backup?
Your shell is probably including the raw wildcards when they don't match
anything, so you should be using include/exclude options instead of the
wildcards. For instance:
rsync -rti --remove-source-files --ignore-existing --include=*.tar --exclude=* \
u...@source:operativo/almacen/{ESP05,EU}/recorte/
/media/BACKUP3_PROMES10/PROMES10km/
..wayne..
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to emulate rdiff behaviour
On Mon, Jun 01, 2009 at 10:15:27AM -0300, Carlos Carvalho wrote: In this case rsync automatically uses --whole-file. Will it do the same in the case of --only-write-batch? The default is always --no-whole-file when writing a batch. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync maintained patches repository
On Thu, May 28, 2009 at 09:04:59PM -0400, Matt McCutchen wrote: How can I access the rsync maintained patches repository with the git tools? Sorry for forgetting to get that taken care of. The issue turned out to be that the samba site defaults to not allowing git: exports unless specifically enabled, and I had neglected to touch the right file to enable access. However, I also just got the repo moved to its own top-level dir: git clone git://git.samba.org/rsync-patches.git That makes its access a little more normal. Access to the old location still works as well (due to a symlink). ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to emulate rdiff behaviour
On Thu, May 28, 2009 at 11:32:16AM -0400, Matt McCutchen wrote: - Each day, generate a reverse batch (by running rsync in the reverse direction with --only-write-batch) and then update the destination file. In order to avoid any inconsistency between the files when the reverse run is done and when the backup copy is done, you could use a second full backup copy (though someone using something like LVM could use snapshots instead). To be explicit, a two-dir method would start by duplicating the backup dir (let's assume to a separate host, though it would also work without bhost: in all these commands): rsync -av /backup/ bhost:/backup2/ Then a backup operation would be: rsync -av srchost:/src/ /backup/ rsync -av --only-write-batch=/batches/$DATE bhost:/backup2/ /backup/ rsync -av /backup/ bhost:/backup2/ That results in the daily double-backup dirs and a batch that lets you get back to the prior day's files. A restore operation would require running a sequence of batch files into the /backup/ dir (without any backups running): rsync -av --read-batch=/batches/$DATE1 /backup/ rsync -av --read-batch=/batches/$DATE2 /backup/ ... Then, after restoring the file(s), revert /backup to the most recent backup data: rsync -av bhost:/backup2/ /backup/ ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync-3.0.6 prints file list generation time wrong ???
On Fri, May 29, 2009 at 06:29:22PM +0530, Jignesh Shah wrote: Does that mean rsync traversed complete 1000k files in 0.671 seconds? Rsync uses an incremental recursion scan by default, so when that is active the time mentioned is just for the getting-started part of the scan. I've been thinking aobut making the sender time each extra scan that it does as the incremental scan progresses and add the times up, but that is not what rsync currently does. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Does rsync detect file corruption?
On Sun, May 24, 2009 at 06:32:40PM +0200, Christian Hecht wrote: Such a tool i plan to write for Mac OS X. The first time it should store checksums and mod times for all files to verify. There are various patches in the patches dir that deal with cached checksums in different ways: - checksum-reading.diff checksum-updating.diff -- both of these use a special file in each dir to store the checksums, with the latter putting code into rsync to be able to update the file when copying the files (both have a perl script to build/update the files and check checksums against the file). - checksum-xattr.diff -- this has a script that puts checksums into an xattr field which rsync can access when it is using the --checksum option. The very latest version of this patch (in git) adds a --list and --check option to the script. - db.diff -- this is the start of some db-accessing code for rsync that can be used to store/access/check checksums (supports sqlite and mysql at present). This is the method I like the best, but the code is in an early state that needs more work. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: patches to enable rsync 3.0.6 compile on AIX 3.2
On Wed, May 13, 2009 at 11:26:32PM -0400, Ian! D. Allen wrote: These two small changes enable 3.0.6 to compile and link on AIX 3.2: #if defined(ENOTSUP) [...] #else errno = EINVAL; I think using ENOSYS would be better than EINVAL there. Does that work? BTW, using -u with diff makes for better patches. Thanks for your feedback! ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[rsync-announce] Rsync 3.0.6 released
I have released rsync 3.0.6. This is another bug-fix release. To see a full summary of the changes since 3.0.5, visit this link: http://rsync.samba.org/ftp/rsync/src/rsync-3.0.6-NEWS You can download the source tar file and its signature from here: http://rsync.samba.org/ftp/rsync/src/rsync-3.0.6.tar.gz http://rsync.samba.org/ftp/rsync/src/rsync-3.0.6.tar.gz.asc The patches directory is in a separate tar file (for those that want one or more of the patches): http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.6.tar.gz http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.6.tar.gz.asc See the rsync website for other download methods, including diffs, etc.: http://rsync.samba.org/ ..wayne.. signature.asc Description: Digital signature ___ rsync-announce mailing list rsync-announce@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-announce
Rsync 3.0.6 released
I have released rsync 3.0.6. This is another bug-fix release. To see a full summary of the changes since 3.0.5, visit this link: http://rsync.samba.org/ftp/rsync/src/rsync-3.0.6-NEWS You can download the source tar file and its signature from here: http://rsync.samba.org/ftp/rsync/src/rsync-3.0.6.tar.gz http://rsync.samba.org/ftp/rsync/src/rsync-3.0.6.tar.gz.asc The patches directory is in a separate tar file (for those that want one or more of the patches): http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.6.tar.gz http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.6.tar.gz.asc See the rsync website for other download methods, including diffs, etc.: http://rsync.samba.org/ ..wayne.. signature.asc Description: Digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Going to upgrad rsync V2.6.7 to V3.0.5
On Thu, May 07, 2009 at 04:50:40PM +0530, Jignesh Shah wrote: I am wondering if we upgrade rsync 3.0.5 then will it cause any unexpected behaviour then rsync 2.6.7? Probably not, but not knowing how you rsync rsync, one can't say for sure. You can read through the various news files, looking for things that may affect you: http://rsync.samba.org/ftp/rsync/src/rsync-2.6.8-NEWS http://rsync.samba.org/ftp/rsync/src/rsync-2.6.9-NEWS http://rsync.samba.org/ftp/rsync/src/rsync-3.0.0-NEWS http://rsync.samba.org/ftp/rsync/src/rsync-3.0.1-NEWS http://rsync.samba.org/ftp/rsync/src/rsync-3.0.2-NEWS http://rsync.samba.org/ftp/rsync/src/rsync-3.0.3-NEWS http://rsync.samba.org/ftp/rsync/src/rsync-3.0.4-NEWS http://rsync.samba.org/ftp/rsync/src/rsync-3.0.5-NEWS http://rsync.samba.org/ftp/rsync/src/rsync-3.0.6-NEWS There is a NOTABLE CHANGES IN BEHAVIOR section in 2 of those files (3.0.0 and 3.0.1). If you run into any problems, you can always try a quick-fix of using either the --no-inc-recursive (--no-i-r) option (which turns off the new, fancy incremental recursion scanning), or the --protocol=29 option (which reverts back to the protocol that 2.6.7 used, which will also disable incremental recursion in addition to several other changes). Most folks are not having problems with the latest rsync, but there are always bugs lying around waiting for less-used use-cases to expose them. I certainly recommend the 3.0.x series over the 2.6.x series. Also I could see so many bug fixes after rsync 3.0.5 release. Do I have to patch all the bug fixes or its fine if I use only rsync 3.0.5? I'd suggest you go with the just-released 3.0.6, personally. :-) ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync docs typo
On Wed, May 06, 2009 at 10:04:40AM -0400, Chris Pepper wrote: the receiving side isn't being running as the receiving side isn't being run as Thanks for pointing that out -- I've checked in a fix. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: %h %a %u string in --log-file-format doesn't work
On Thu, May 07, 2009 at 04:42:28PM +0530, Jignesh Shah wrote: It does't include the host name, IP address and authenticated user name and instead it keep it as %h %a %u. Could any one tell me what is wrong here? You aren't doing a daemon copy, which is the only time those values are currently defined. I'm considering enhancing %a and %h to derive the IP from the environment when possible (like daemon-mode over remote-shell currently does). That would let someone do normal (non-daemon), ssh copies and still get an IP+hostname put into the log file. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] allow to disable SLP with runtime option
On Tue, Apr 28, 2009 at 10:00:21AM +0200, Petr Uzel wrote: I wanted to see changes you've made, but I can't figure out how to 'git clone' the patches repository. Samba doesn't allow cloning of the patches repo at this time, but it's something I hope to get fixed before too long. To view patches, go to the download page and look for the patches section. http://rsync.samba.org/download.html ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Code study]append_mode definitions
On Mon, Apr 27, 2009 at 10:33:02AM +0800, Daniel.Li wrote: Can anyone help to explain definitions of variable append_mode? There are two different type of appending (see the man page) depending on how new rsync is and how much of the file should be checksummed. The option is also disabled if a file is being fixed during a redo event. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync: writefd_unbuffered failed to write 4 bytes on mounted share
On Mon, Apr 27, 2009 at 12:19:10PM +0200, Stefan-Michael Guenther wrote: rsync: close failed on /media/nas/daily.0/samba/Bereiche/Auslieferung/Software/NT2/2012/2012-69-RA-1L.0968 (XY_I)/2006-11-16/XY_app.968/Logdb/.XYlog.mdb.REP380: Input/output error (5) That's an OS error when rsync closed a file it was writing, so you'll need to figure out what your OS is telling you about that failure on that disk. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: purge-empty-dirs and max-file-size confusion
On Fri, Apr 24, 2009 at 02:19:42PM -0400, Ian! D. Allen wrote: There is no mention of the concept of transfer rule in the rsync man page. I offer some proposed man page wording changes, below. Thanks. I have committed some manpage changes that clarify this unexpected behavior. At some point rsync may allow actual filtering of files by their (non-name) attributes, which would avoid this situation. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] allow to disable SLP with runtime option
On Wed, Apr 22, 2009 at 04:14:23PM +0200, Petr Uzel wrote: It adds a new global boolean option, 'disable slp', which can be used to disable SLP advertisements at runtime. I've taken your patch and changed the boolean into a use slp global daemon parameter (default: enabled). I also made sure that if slp is disabled, that there is no slp timeout processing (or services could get enabled at the first timeout). The updated slp patch is now in git. Thanks! ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Error messages 3.0.6pre1
On Fri, Apr 17, 2009 at 03:03:02PM -0400, Warren Oates wrote: I compiled 3.0.6pre1, including the crtimes and fileflags patches for use on Intel OS X 10.5.6 I just compiled rsync with those patches on an Intel OS X 10.5.6 system, and didn't see any such errors. I did see an issue with make check (an output problem in the daemon test's file listing) which pointed me at an issue with --list-only's handling of create times. I've fixed that for the next release. Do you see free warnings for all rsync copies? What does file rsync say for the rsync binary? I'd suggest building your own rsync, and if the errors continue, I'll need some help to figure out what free is causing them. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and quota in Linux
On Thu, Apr 16, 2009 at 02:49:35PM +0100, Joao Ferreira wrote: can I expect the quotas to be updated correctly on the receiving side ? Quota is handled by your OS, so is out of rsync's hands. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Unable to transfer between win32 (interix) and FreeBSD
On Thu, Apr 16, 2009 at 03:53:24PM +0100, Steven Hartland wrote: unexpected tag -96 [receiver] You don't mention what options you're using, so I'm unable to try to hint you towards option tweaks to try. e.g. if using -z, try dropping it. You could also try forcing an older protocol (e.g. --protocol=29) to try to help narrow down the issue. The error above indicates that the binary protocol is out of whack. That either means that rsync is confused about what data comes next (which could be the writer or the reader's fault), or that your connection corrupted the bytes. There is a way to make a record of the data that is being sent and is received to check if it differs by using the savetransfer program (see the C source in the support dir for its usage). ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Buffer Cache Thrashing
On Thu, Apr 16, 2009 at 10:44:29PM -0400, Josh Snyder wrote: Rsync thrashes my buffer cache when transfering large files. I see that some people worked on this problem about a year ago. Did they find a workable solution? Some Linux-only work was done, but the patches really revealed that there needs to be a much better way to ask Linux to not fill up the cache with the current program's workload since the solution turned into a pretty big kluge to try to get Linux to do the right thing. It would be cool if some improvements were done at the OS level to fix the issue without requiring individual programs to be hacked up. However, work could be done to cleanup the prior cache-affecting work and merge together the current patches/drop-cache.diff code with the later improvements from Tobi Oetiker, but cleaned up to be less intrusive in the code (e.g. to use more macros instead of tweaking lots of system calls, which makes it easier to maintain). ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Excluded directories at rsync (was --filter, missing in action)
On Thu, Apr 23, 2009 at 08:40:12PM +0300, Teodor MICU wrote: This is not desired because these parent* directories were deleted on the MASTER host but cannot be deleted with rsync on the SLAVE hosts because it contains one excluded directory named DIR. You should either anchor your excludes so they only match the items you want them to (not extra subdirs), or you should use the perishable option to the exclude rule so that it won't affect the removal of a parent directory. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: looking for superlifter souce code and related information
On Tue, Apr 21, 2009 at 10:39:58AM +0800, Daniel.Li wrote: I just googled superlifter and found below link, but I can't get any source code. Sadly, superlifter never achieved lift-off, so there was never any source code for it. I created my own test-bed for some new-protocol experiments (which you cite in your follow-up -- rZync), but after that I decided to work on adding incremental scanning to rsync in way that would keep the program backward-compatible with older rsync versions. I may finally revisit superlifter/rZync at some point here, but haven't had the time for that lately. Are other folks wanting to work on ideas for superlifter? ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Any program flow or software spec for Rsync
On Tue, Apr 21, 2009 at 02:47:41PM +0800, Daniel.Li wrote: I'm trying to take a look at rsync from code-level. Is there any software spec or program flow that I can get, which might give me a basic understanding about the code? The only thing I can think of is an old how-rsync-works page: http://rsync.samba.org/how-rsync-works.html Beyond that, rsync's protocol and/or program flow is not documented anywhere. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --filter, missing in action.
On Thu, Apr 23, 2009 at 02:02:26PM +0200, istrice wrote: --filter=':r .bar' This indicates that you want the file .bar to be read at every level of the destination hierarchy and its contents to be interpreted as filter commands. That would let you specify localized filtering. --filter=':r foo/' It doesn't make sense to specify a directory as a merge-file. From your other text I imagine you're trying to protect the foo dir's contents from deletion, in which case you should be using a protect (P) rule: --filter='protect foo/' ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: purge-empty-dirs and max-file-size confusion
On Wed, Apr 22, 2009 at 02:20:37AM -0400, Ian! D. Allen wrote: I want to use --min-size to copy just large files (and their necessary parent directories), but everything I've tried copies *all* the source directories, and creates them empty on the destination even if they don't have any big files in them. I only want the minimal directory hierarchies that contain the big files. This is because --min-size is a transfer rule, not an exclude rule. An exclude rule would affect deletions, and --min-size just affects what is transferred out of the full set of files that are present. Thus, the dirs with smaller files are not actually empty, they just don't have any files that match the transfer rule. There is not currently a way include/exclude files based on size in rsync. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Overwrite the rsync log-file in rsync3.0.5 ??
On Fri, Apr 17, 2009 at 02:26:25PM +0530, Jignesh Shah wrote: Hi, Could any one please let me know if there is anyway to overwrite the log-file if it already exists. Just change your script that runs rsync to do a rm /tmp/mylogs/myfile or cp /dev/null /tmp/mylogs/myfile prior to the transfer. Rsync always appends. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync 3.0.6pre1 released
Rsync version 3.0.6pre1 is now available for release testing. This is a bug-fix release. Please test this new release and send email to the rsync mailing list with any questions, comments, or bug reports. To see a full summary of the changes since 3.0.5, visit this link: http://rsync.samba.org/ftp/rsync/src-previews/rsync-3.0.6pre1-NEWS You can download the source tar file and its signature from here: http://rsync.samba.org/ftp/rsync/src-previews/rsync-3.0.6pre1.tar.gz http://rsync.samba.org/ftp/rsync/src-previews/rsync-3.0.6pre1.tar.gz.asc The patches directory is now in a separate tar file (for those that want one or more of the patches): http://rsync.samba.org/ftp/rsync/src-previews/rsync-patches-3.0.6pre1.tar.gz http://rsync.samba.org/ftp/rsync/src-previews/rsync-patches-3.0.6pre1.tar.gz.asc See the rsync website for other download methods, including diffs, etc.: http://rsync.samba.org/ ..wayne.. signature.asc Description: Digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 3.0.4 ACL corruption
On Mon, Dec 15, 2008 at 12:20:23PM +0100, Peter Rindfuss wrote: I've just noticed that ACLs get partially corrupted on the receiving side. This is my command line, running as root: /usr/bin/rsync --quiet --links --numeric-ids --acls --perms --times --recursive --owner --group --delete-during --ignore-errors --backup --backup-dir=/wzb/backup/wzb= --password-file=/etc/wzb/rsync/password --filter '- /backup/' rsync://selene.wzb.eu/wzb /wzb This was caused by the backup option putting extra entries into the ACL cache, which caused the cache to get out of sync with the sender's numbering scheme. It could also affect xattrs. I've just checked-in a fix that took care of the issue in a test setup where I replicated the problem. The fix will go out in 3.0.6. Thanks for the report! ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem with extended ACLs in 3.0.4?
On Sun, Nov 02, 2008 at 07:18:39PM +, Andrew Gideon wrote: The previous copy of the file has the correct/complete ACL, and the link-dest logic sees this as different from the new copy result so a new copy of the file - with the wrong ACL - is written. Rsync was of the belief that a mask was only needed if an ACL had named values, otherwise it tried to simplify the ACL to mask off the group mode and dropped the mask. I've checked in a change that makes it keep whatever mask value is specified, so the ACLs should be identical now. This will get released in 3.0.6. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: itemized option (-ii) with --log-file and --log-file format
On Wed, Apr 08, 2009 at 07:41:12AM +0530, Jignesh Shah wrote: No, I also want to log files that are not transferred. You can use -vv instead of -ii to get a list of uptodate items, but no mention of attribute changes (non-transferred files are all uptodate). In rsync 3.1.0dev you can get that info without all the rest of the verbosity items it implies by specifying --info=name2. If that is not what you want, then you need the %i field so that rsync can output what has changed (or not changed) for the current item. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: itemized option (-ii) with --log-file and --log-file format
On Mon, Apr 06, 2009 at 10:45:42AM +0530, Jignesh Shah wrote: I have to use -ii so that it print all the statistic info about what files/dirs synched and what are not. No, -ii will include unchanged files in the logs in addition to changed files, and thus a %i field (the one with the cryptic characters) is needed to distinguish the two. The -i option also tells rsync to log about files that have attribute changes (not just transferred files), and the %i field is necessary to be able to distinguish if a file just got tweaked or was transferred. If you just want to log transferred files, don't specify -i at all, and use your prior --log-file=FILE and --log-file-format=FORMAT options, but leave out the %i parameter in the latter. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Does rsync use md4 or md5?
On Mon, Apr 06, 2009 at 12:27:38PM +0200, David Depardoux wrote: I guess the man page is only a bit outdated? Yeah -- thanks for pointing that out. I've checked-in a fix to the git repository. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --times does not transfer full modification time
On Mon, Apr 06, 2009 at 12:28:50PM +0200, David Depardoux wrote: Thus, the milliseconds for files on the receiving side are all set to zero. Correct. Rsync does not (yet?) support sub-second timestamps. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: itemized option (-ii) with --log-file and --log-file format
On Wed, Apr 01, 2009 at 10:50:13AM +0530, Jignesh Shah wrote: I noticed that the output format for this command is different from rsync V2.6.* . Is there any way I can make this output similar to rsync V2.6.*? What difference are you referring to? The use of send instead of recv for a local transfer? That is a more accurate representation of what rsync is doing, but you can change it by forcing rsync to pull files from localhost. See the support/lsh script for a way to pull from localhost w/o using a remote shell. If that's not the issue, please elaborate. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: verbosity: list files without summary?
On Tue, Mar 31, 2009 at 10:43:51AM +0200, Mi wrote: Is there a way to set the verbosity so that transferred/changed files are listed, but not the summary information. See the --out-format (aka --log-format in older rsync versions) and the -i (--itemize-changes) options (since -i is an easy way to set --log-format and get extra information on what is changing). ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Why are files synced again and again when not using -a?
On Mon, Mar 30, 2009 at 02:13:49PM +0200, Andr? H?nsel wrote: usually I use rsync with -a but recently I used it without this option and discovered that the same files are synced over and over again (as shown by -v). You should at least specify -t when not using -a. See the -a option in the manpage to see what options it implies. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: ssh rsync equivalence?
On Sun, Mar 29, 2009 at 05:15:58PM +1300, Morgan Read wrote: ssh [...] sudo rsync [...] -e 'ssh -i /home/rsync/.ssh/id_dsa' [...] Those quotes won't survive the trip through that many programs, so you're undoubtedly ending up with rsync seeing just the -e ssh option, and the -i and beyond are taken as options/args to rsync itself. Your best bet is to put ssh config items into your /root/.ssh/config file. For instance: Host 192.168.1.100 User root IdentityFile /home/rsync/.ssh/id_dsa That sets those settings as the default when accessing that IP. Or give it a name and use that name instead of a hostname or IP when telling rsync what host to contact. For instance, in the local host's .ssh/config file: Host r40 Host 192.168.1.40 User rsync Then on the remote host: Host r100 Host 192.168.1.100 User root IdentityFile /home/rsync/.ssh/id_dsa Then run this (trailing slash added to avoid /path/etc/etc): ssh -t r40 sudo rsync -avzAXHn --delete-after /etc/ r100:/path/etc Your other option is to create a script that executes ssh with the option(s) you want to use and run that script via rsync's -e option. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Protocol of checksums generated with -c option?
On Sat, Mar 21, 2009 at 02:06:42PM +0100, Christian Hecht wrote: Now my question is, when i use rsync with the -c option, is it possible to output the filename and the generated hash to any file? Rsync 3.1.0dev supports the %C escape in log formats, which can output the sender's checksum anytime a file is transferred (even without -c), or for any file when -c is used. Example options for controlling the output format: --out-format='%C %n%L' -ii --out-format='%i %C %n%L' There is also a log-checksum.diff in the patches directory for the 3.0.x releases. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Weird rsync or shell problem?
On Sat, Mar 21, 2009 at 07:23:58PM +0100, axel.we...@cbc.de wrote: SCHNITT/Render Files/#342#200#242Vorlage Bleskin_990101_NB_n-tv/Vorlage Blesk-FIN-000b The character sequence 0343 0200 0242 is E2 80 A2 in hex. Seems to be a common sequence for something (since those characters show up via google search), but I'm not sure what they are (and you don't mention your system's character encoding). The difference in behavior between cron and shell is most likely that the environment variable LANG is set differently, and thus rsync doesn't think that the character sequence is valid for the current language (and thus it escapes it). ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: AARGH! I can't make rsync not copy dirs with --files-from option
On Sat, Mar 14, 2009 at 04:11:12PM -0500, David C. Rankin wrote: Way too much path, way way too much path. By default, all the path segments in the files-from file are a part of the transfer unless you turn off the implied --relative option, in which case no directory elements from the source paths are used. So, either (1) move path elements out of the files-from file to the source dir arg, (2) specify --no-relative and specify the full destination path to where you want the directory files to go, or (3) insert a /./ dir to specify the cut point for files-from path information (but prior to 3.0.5 the sender needs to be in the same dir as the source path to avoid a bug). So, option 1 would mean stripping things like /var/cache/zypp/packages from the start of the files-from paths and specifying that source path on the command line as /var/cache/zypp/packages instead of /. In option 3, files-from lines would look like this: /var/cache/zypp/packages/openSUSE-11.0-Updates/rpm/./x86_64/satsolver-tools-0.9.5_0.9.6-0.1.x86_64.delta.rpm Put that '/./' dir wherever you want to cut off the path info that is included in the transfer. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync flags freebsd
On Thu, Mar 12, 2009 at 10:35:45PM -0400, Matt McCutchen wrote: The rsync 3.0.5 version of this patch is at: http://rsync.samba.org/ftp/rsync/patches/fileflags.diff I've just revised how patches are stored so that I can have patches for 3.0.x as well as 3.1.0dev. Thus, the link you cited is no longer there. In the new setup, use gitweb to view the 3.0.x branch of the patches.git repository and click on the raw link for the file you want. The patches.git repo is here: http://gitweb.samba.org/?p=rsync.git/patches.git And a tree-view of its 3.0.x branch is here: http://gitweb.samba.org/?p=rsync.git/patches.git;a=tree;h=refs/heads/b3.0.x;hb=refs/heads/b3.0.x ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync flags freebsd
On Fri, Mar 13, 2009 at 06:57:15PM -0400, Matt McCutchen wrote: I would have thought it more natural to have branches patches-master and patches-3.0.x in the main repository. Is there any reason you made a separate repository? Yes, this way the branch names match and I can tag releases. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: comments with '#' in --files-from
On Mon, Mar 09, 2009 at 01:36:26AM -0400, Matt McCutchen wrote: From the code, it looks like comments (as lines beginning with '#' or ';') are supported but are buggy on remote runs (probably what you are seeing). Yeah, the code is supposed to ignore comments unless the data is coming in via the socket, in which case the remote side will have already removed them. A file being read remotely is probably being mis-parsed as though it was coming in via the socket. The way to add a file that starts with a '#' or a ';' (that also works for any file) is to start the filename with a slash, because a '/' means the root of the source dir in a files-from file. So, a simple rule of always start your file paths with '/' (or even ./) would make adding comments safe without the risk of misparsing a name. Your idea to disallow comments if --from0 is specified seems reasonable to me, since the user is asking for the ability to store even newlines in a filename, it makes sense that the names should also be pre-filtered of any comments too. That should simplify the code, since the client will have sent the server a --from0 option when it is sending the names via the socket. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: proposal to speed rsync with lots of files
On Thu, Mar 05, 2009 at 03:27:50PM -0800, Peter Salameh wrote: My proposal is to first send a checksum of the file list for each directory. If is found to be identical to the same checksum on the remote side then the list need not be sent for that directory! My rZync source does something like that for directories: it treats a directory-list transfer like a file transfer. That means that the receiving side sends a set of checksums to the sending side telling it what it's version of the directory looks like, and then the sender sends a normal set of delta data that lets the receiver reconstruct the sender's version of the directory (which it compares to its own). One potential drawback is having to deal with false checksum-matches (which should be rare, but would require the dir data to be resent) I hadn't optimized it for block size or (possibly) data order to make it more efficient, but it is an interesting idea for speeding up a slow connection. I'm not sure if it would really help out that much for a more modern, faster connection, because rsync sends the file-list data at the same time as it is being scanned, and sometimes the scan is the bottle-neck. The best way to optimize sending of really large numbers of files that are mostly the same is to start to leverage a file-change notification system, such as inotify. Using that, it is possible to distill a list of what files/directories need to be copied, and to just copy what is needed. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problems while transferring big files
On Fri, Mar 06, 2009 at 08:20:43AM +0100, Boniforti Flavio wrote: I'm using rsync -rtz to transfer over an SSH tunnel and I always get 12 Error in rsync protocol data stream That is just the error code and its interpretation. What error message comes prior to that? We hypothesize that there can be an accidental match in the checksum data, which would cause the two sides to put different streams of data into their gzip compression algorithm, and eventually get out of sync and blow up. If you have a repeatable case of a new file overwriting an existing file that always fails, and if you can share the files, make them available somehow (e.g. put them on a web server) and send the list (or me) an email on how to grab them, and we can run some tests. If the above is the cause of the error, running without -z should indeed avoid the issue. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Delete files specified in --from-files-file that don't exist on the sender side
On Fri, Mar 06, 2009 at 04:44:56PM -0500, Matt McCutchen wrote: Two issues you may wish to correct [...] Thanks for pointing those out! I've made sure that the mode does not get tweaked by --chmod if it is 0, and changed the --list-only output to start with *missing, as you suggested. While you're hacking on those areas of code, perhaps you would consider adding a --sources-if-exist option to treat the nonexistence of a source argument or --files-from entry as not an error. If someone is doing inotify-based --files-from transfers and a filename is now missing, maybe they really want --delete-missing-args to ensure that the receiver doesn't have a version of the now-missing file. If not, I think --ignore-missing-args might be a nice matching option name. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --files-from
On Thu, Feb 26, 2009 at 07:55:22PM -0700, Manoj Joseph wrote: Is there a way of passing a list of source _and_ destination paths?? Only for a single-file transfer. So, to do this with rsync, you'd need to transfer each file separately, possibly using the lighter daemon protocol (which could be routed over an established ssh tunnel). The librsync project on sourceforge (librsync.sourceforge.net) seems to be unmaintained. Does anyone on this list have any experience with using rsync as a library? To see an example of using librsync to build an rsync-like tool, see my rZync code: http://opencoder.net/new-protocol.html This tool supports being controlled by a series of transfer commands, much like an sftp that can use the rsync delta-transfer protocol. The code was never fully polished (e.g. the uid/gid code needs work), but you might find it useful in either making something simpler, or in testing/debugging/enhancing what is there. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Treat files that were modified locally and remote
On Sat, Feb 28, 2009 at 01:31:32PM +0100, Kurt wrote: One way to identify such files would be to do a dry-run upsync, then a dry-run downsync, find files that would have been sync'ed in both dry-runs, and then prompt the user for some action. Assuming that you mean to use the -u option, that still wouldn't work very well because you don't know if the updated file is newer than the original file, or a newer change than the also-changed file on the other side. You need something with a DB, such as unison, or (perhaps) the drsync perl wrapper script. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Delete files specified in --from-files-file that don't exist on the sender side
On Fri, Feb 13, 2009 at 12:31:54AM +0100, intripoon wrote: Now there are files I want to have deleted on PC B. They don't exist on PC A. I hoped I could just add them to the list of files given to --from-files, but the sender just tells me failed: No such file or directory (2) and goes on. I added an option to the latest development version that supports this: --delete-missing-args. Can be used with or without --files-from. See the git repository or the latest nightly tar file if you'd like to try it out. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 3.0.5 - ERROR: buffer overflow in recv_file_entry [generator]
On Mon, Feb 16, 2009 at 02:54:36PM +0100, dr...@wp.pl wrote: overflow: xflags=0x7a l1=255 l2=779 This is the error rsync gives when the file and path exceed the MAXPATHLEN size for your system. The receiver updates all the files from the base of the transfer, so the names need to fit within the maximum size that your OS's system calls can handle, or rsync dies. The numbers show that your MAXPATHLEN is less than 779+255, so it's probably set to an unusually small 1024. If your system can actually handle longer paths, recompile rsync with a better MAXPATHLEN size. In rsync.h, MAXPATHLEN is defined to 1024 if the value is not defined at all, so try increasing that value (try 4096). If instead your OS defines the size to 1024, you can try adding an #undef MAXPATHLEN before the #ifndef MAXPATHLEN line and see if things work with a forced value. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Renamed files and directories
On Mon, Feb 16, 2009 at 10:14:25AM +0100, N.J. van der Horn (Nico) wrote: What is the current status of both rename-patches ? Are there alternative measures ? I'm not thrilled with how the rename patches work, especially since they disable incremental recursion. As such, I'm hoping to change how they work before considering adding renaming handling to a released version. One option is making further enhancements to the db.diff patch that would let rsync leverage DB info on where files are and what their checksums are. If the DB were kept updated via something like inotify, it would be an especially nice way to quickly find a file with the right size+mtime+checksum (or user-selected attributes). For those without inotify, an upgrade script could be run, perhaps right before the transfer. Or perhaps rsync could leverage one of the existing file-monitoring packages -- does something like beagle keep track of the file's checksum? ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: uid/gid settings in rsyncd.conf not respected?
On Sat, Feb 14, 2009 at 08:53:22PM -0500, Matt McCutchen wrote: The attached patch (also in wip/supplementary-groups of my repository) adds a daemon parameter to take on the supplementary groups. I went a little different route than this path by allowing the user to specify one or more groups via the gid setting. It will also expand the string * (if specified as the first item) into the normal grouplist for the requested user. I then changed a non-super-user daemon-run to default the uid/gid parameters to NULL, which allows rsync to know if the user requested a value, and it will now complain if a specified setting fails. A super-user run still defaults to nobody if unspecified. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --password-file
On Sun, Feb 15, 2009 at 05:08:36AM -0700, lewis butler wrote: This tells everyone the exact length of each password The docs should actually say that everything after the first line of the file is ignored. So, feel free to add any text you like after that. I'll improve the man page to get this right. First off, it would let you have multiple passwords in a single file That would make some things easier, but I'd want any such change to be backward compatible with current files. One possibility it to leave the command-line option alone and adding support for a new environment variable, such as RSYNC_PASSWORDS_FROM. That would let the new-style file be specified in a shell rc file and used to configure multiple servers/modules, as desired. I'd at the very least include such a change in the patches directory if someone wanted to code something like that. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
