The branch, master has been updated via 9cd85b84 Skip an append if sender's file gets shorter. via f8dcd7d4 Improve the docs for `--archive`. via 69530b40 Avoid output variance in protocol 29. via 122b0fdc Check status of tests that pipe rsync's output & simplify output diffing. via b990d97d Put CAN_HARDLINK_SYMLINK info into --version output. via fd6839b7 Avoid spurious "is newer" messages with `--update`. from a79d9b22 Update the NEWS.
https://git.samba.org/?p=rsync.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 9cd85b849672c93c4b1f534898585f92ec4de30d Author: Wayne Davison <wa...@opencoder.net> Date: Mon Sep 21 14:40:23 2020 -0700 Skip an append if sender's file gets shorter. Fixes bug #90. Similar to a pull request by Tomas Korbar. commit f8dcd7d452b137fa49b71e714512e7d619a4d0dd Author: Wayne Davison <wa...@opencoder.net> Date: Mon Sep 21 14:05:51 2020 -0700 Improve the docs for `--archive`. A slightly tweaked version of a patch from Richard Michael. commit 69530b406e51ef9c50186611ddd181533eb2101f Author: Wayne Davison <wa...@opencoder.net> Date: Mon Sep 21 13:45:42 2020 -0700 Avoid output variance in protocol 29. commit 122b0fdc4f1fb9595f633fd4daf347de157ff659 Author: Wayne Davison <wa...@opencoder.net> Date: Mon Sep 21 13:15:43 2020 -0700 Check status of tests that pipe rsync's output & simplify output diffing. commit b990d97d35c39a1bf508b9819c47b0876686f333 Author: Wayne Davison <wa...@opencoder.net> Date: Mon Sep 21 13:17:15 2020 -0700 Put CAN_HARDLINK_SYMLINK info into --version output. commit fd6839b7464f53afb4cf016e4b4f6d2c477eb8d8 Author: Wayne Davison <wa...@opencoder.net> Date: Mon Sep 21 10:44:18 2020 -0700 Avoid spurious "is newer" messages with `--update`. ----------------------------------------------------------------------- Summary of changes: NEWS.md | 8 +++ generator.c | 2 +- rsync.1.md | 16 +++--- sender.c | 10 ++++ testsuite/backup.test | 5 +- testsuite/daemon.test | 29 +++------- testsuite/devices.test | 31 +++-------- testsuite/duplicates.test | 2 - testsuite/exclude.test | 18 +++---- testsuite/hardlinks.test | 4 +- testsuite/itemize.test | 133 ++++++++++++---------------------------------- testsuite/relative.test | 2 - testsuite/rsync.fns | 90 ++++++++++++++++++++++++------- usage.c | 19 ++++--- 14 files changed, 171 insertions(+), 198 deletions(-) Changeset truncated at 500 lines: diff --git a/NEWS.md b/NEWS.md index 0d65d2e5..405e86d1 100644 --- a/NEWS.md +++ b/NEWS.md @@ -9,6 +9,14 @@ - Fix a bug with `--mkpath` if a single-file copy specifies an existing destination dir with a non-existing destination filename. + - Fix `--update -vv` to output "is uptodate" instead of "is newer" messages + for files that are being skipped due to an identical modify time. (This + was a new output quirk in 3.2.3.) + + - When doing an append transfer, the sending side's file must not get shorter + or it is skipped. Fixes a crash that could occur when the size changes to 0 + in the middle of the send negotiations. + - Avoid a weird failure if you run a local copy with a (useless) `--rsh` option that contains a `V`. diff --git a/generator.c b/generator.c index f1780838..f83ac501 100644 --- a/generator.c +++ b/generator.c @@ -1703,7 +1703,7 @@ static void recv_generator(char *fname, struct file_struct *file, int ndx, goto cleanup; } - if (update_only > 0 && statret == 0 && file->modtime - sx.st.st_mtime <= modify_window) { + if (update_only > 0 && statret == 0 && file->modtime - sx.st.st_mtime < modify_window) { if (INFO_GTE(SKIP, 1)) rprintf(FINFO, "%s is newer\n", fname); #ifdef SUPPORT_HARD_LINKS diff --git a/rsync.1.md b/rsync.1.md index 62b810e2..7bb4c5a1 100644 --- a/rsync.1.md +++ b/rsync.1.md @@ -338,7 +338,7 @@ detailed description below for a complete description. --quiet, -q suppress non-error messages --no-motd suppress daemon-mode MOTD --checksum, -c skip based on checksum, not mod-time & size ---archive, -a archive mode; equals -rlptgoD (no -H,-A,-X) +--archive, -a archive mode is -rlptgoD (no -A,-X,-U,-N,-H) --no-OPTION turn off an implied OPTION (e.g. --no-D) --recursive, -r recurse into directories --relative, -R use relative path names @@ -711,12 +711,12 @@ your home directory (remove the '=' for that). 0. `--archive`, `-a` This is equivalent to `-rlptgoD`. It is a quick way of saying you want - recursion and want to preserve almost everything (with `-H` being a notable - omission). The only exception to the above equivalence is when - `--files-from` is specified, in which case `-r` is not implied. + recursion and want to preserve almost everything. Be aware that it does + **not** include preserving ACLs (`-A`), xattrs (`-X`), atimes (`-U`), + crtimes (`-N`), nor the finding and preserving of hardlinks (`-H`). - Note that `-a` **does not preserve hardlinks**, because finding - multiply-linked files is expensive. You must separately specify `-H`. + The only exception to the above equivalence is when + `--files-from` is specified, in which case `-r` is not implied. 0. `--no-OPTION` @@ -949,7 +949,9 @@ your home directory (remove the '=' for that). existing content in the file (it only verifies the content that it is appending). Rsync skips any files that exist on the receiving side that are not shorter than the associated file on the sending side (which means - that new files are trasnferred). + that new files are trasnferred). It also skips any files whose size on the + sending side gets shorter during the send negotiations (rsync warns about a + "diminished" file when this happens). This does not interfere with the updating of a file's non-content attributes (e.g. permissions, ownership, etc.) when the file does not need diff --git a/sender.c b/sender.c index 94761c26..9cfca134 100644 --- a/sender.c +++ b/sender.c @@ -362,6 +362,16 @@ void send_files(int f_in, int f_out) exit_cleanup(RERR_FILEIO); } + if (append_mode > 0 && st.st_size < F_LENGTH(file)) { + rprintf(FWARNING, "skipped diminished file: %s\n", + full_fname(fname)); + free_sums(s); + close(fd); + if (protocol_version >= 30) + send_msg_int(MSG_NO_SEND, ndx); + continue; + } + if (st.st_size) { int32 read_size = MAX(s->blength * 3, MAX_MAP_SIZE); mbuf = map_file(fd, st.st_size, read_size, s->blength); diff --git a/testsuite/backup.test b/testsuite/backup.test index 87db33e5..3b49db94 100644 --- a/testsuite/backup.test +++ b/testsuite/backup.test @@ -15,8 +15,6 @@ makepath "$fromdir/deep" "$bakdir/dname" name1="$fromdir/deep/name1" name2="$fromdir/deep/name2" -outfile="$scratchdir/rsync.out" - cat "$srcdir"/[gr]*.[ch] > "$name1" cat "$srcdir"/[et]*.[ch] > "$name2" @@ -26,8 +24,7 @@ checkit "$RSYNC -ai --info=backup '$fromdir/' '$chkdir/'" "$fromdir" "$chkdir" cat "$srcdir"/[fgpr]*.[ch] > "$name1" cat "$srcdir"/[etw]*.[ch] > "$name2" -$RSYNC -ai --info=backup --no-whole-file --backup "$fromdir/" "$todir/" \ - | tee "$outfile" +checktee "$RSYNC -ai --info=backup --no-whole-file --backup '$fromdir/' '$todir/'" for fn in deep/name1 deep/name2; do grep "backed up $fn to $fn~" "$outfile" >/dev/null || test_fail "no backup message output for $fn" diff $diffopt "$fromdir/$fn" "$todir/$fn" || test_fail "copy of $fn failed" diff --git a/testsuite/daemon.test b/testsuite/daemon.test index 836ce33b..80d2baf6 100644 --- a/testsuite/daemon.test +++ b/testsuite/daemon.test @@ -21,9 +21,6 @@ . "$suitedir/rsync.fns" -chkfile="$scratchdir/rsync.chk" -outfile="$scratchdir/rsync.out" - SSH="src/support/lsh.sh --no-cd" FILE_REPL='s/^\([^d][^ ]*\) *\(..........[0-9]\) /\1 \2 /' DIR_REPL='s/^\(d[^ ]*\) *[0-9][.,0-9]* /\1 DIR /' @@ -57,21 +54,17 @@ test-to r/w test-scratch NOCOMMENT EOT -$RSYNC -ve "$SSH" --rsync-path="$RSYNC$confopt" localhost:: | tee "$outfile" +checkdiff2 "$RSYNC -ve '$SSH' --rsync-path='$RSYNC$confopt' localhost::" echo '====' -diff $diffopt "$chkfile" "$outfile" || test_fail "test 0 failed" RSYNC_CONNECT_PROG="$RSYNC --config=$conf --daemon" export RSYNC_CONNECT_PROG -$RSYNC -v localhost:: | tee "$outfile" +checkdiff2 "$RSYNC -v localhost::" echo '====' -diff $diffopt "$chkfile" "$outfile" || test_fail "test 1 failed" -$RSYNC -r localhost::test-hidden \ - | sed "$FILE_REPL" | sed "$DIR_REPL" | sed "$LS_REPL" \ - | tee "$outfile" -cat <<EOT >"$chkfile" +checkdiff "$RSYNC -r localhost::test-hidden" \ + "sed -e '$FILE_REPL' -e '$DIR_REPL' -e '$LS_REPL'" <<EOT drwxr-xr-x DIR ####/##/## ##:##:## . drwxr-xr-x DIR ####/##/## ##:##:## bar -rw-r--r-- 4 ####/##/## ##:##:## bar/two @@ -80,24 +73,18 @@ drwxr-xr-x DIR ####/##/## ##:##:## bar/baz drwxr-xr-x DIR ####/##/## ##:##:## foo -rw-r--r-- 4 ####/##/## ##:##:## foo/one EOT -diff $diffopt "$chkfile" "$outfile" || test_fail "test 2 failed" -$RSYNC -r localhost::test-from/f* \ - | sed "$FILE_REPL" | sed "$DIR_REPL" | sed "$LS_REPL" \ - | tee "$outfile" -cat <<EOT >"$chkfile" +checkdiff "$RSYNC -r localhost::test-from/f*" \ + "sed -e '$FILE_REPL' -e '$DIR_REPL' -e '$LS_REPL'" <<EOT drwxr-xr-x DIR ####/##/## ##:##:## foo -rw-r--r-- 4 ####/##/## ##:##:## foo/one EOT diff $diffopt "$chkfile" "$outfile" || test_fail "test 3 failed" if $RSYNC --version | grep "[, ] atimes" >/dev/null; then - $RSYNC -rU localhost::test-from/f* \ - | sed "$FILE_REPL" | sed "$DIR_REPL" | sed "$LS_REPL" \ - | tee "$outfile" - cat <<EOT >"$chkfile" + checkdiff "$RSYNC -rU localhost::test-from/f*" \ + "sed -e '$FILE_REPL' -e '$DIR_REPL' -e '$LS_REPL'" <<EOT drwxr-xr-x DIR ####/##/## ##:##:## foo -rw-r--r-- 4 ####/##/## ##:##:## ####/##/## ##:##:## foo/one EOT - diff $diffopt "$chkfile" "$outfile" || test_fail "test 3 failed" fi diff --git a/testsuite/devices.test b/testsuite/devices.test index 56ab2c52..79dfdf17 100644 --- a/testsuite/devices.test +++ b/testsuite/devices.test @@ -9,9 +9,6 @@ . "$suitedir/rsync.fns" -chkfile="$scratchdir/rsync.chk" -outfile="$scratchdir/rsync.out" - # Build some hardlinks case $0 in @@ -116,33 +113,21 @@ mkfifo "$fromdir/fifo" || mknod "$fromdir/fifo" p || test_skipped "Can't run mkf # Work around time rounding/truncating issue by touching both files. touch -r "$fromdir/block" "$fromdir/block" "$fromdir/block2" -$RSYNC -ai "$fromdir/block" "$todir/block2" \ - | tee "$outfile" -cat <<EOT >"$chkfile" +checkdiff "$RSYNC -ai '$fromdir/block' '$todir/block2'" <<EOT cD$all_plus block EOT -diff $diffopt "$chkfile" "$outfile" || test_fail "test 1 failed" -$RSYNC -ai "$fromdir/block2" "$todir/block" \ - | tee "$outfile" -cat <<EOT >"$chkfile" +checkdiff "$RSYNC -ai '$fromdir/block2' '$todir/block'" <<EOT cD$all_plus block2 EOT -diff $diffopt "$chkfile" "$outfile" || test_fail "test 2 failed" sleep 1 -$RSYNC -Di "$fromdir/block3" "$todir/block" \ - | tee "$outfile" -cat <<EOT >"$chkfile" +checkdiff "$RSYNC -Di '$fromdir/block3' '$todir/block'" <<EOT cDc.T.$dots block3 EOT -diff $diffopt "$chkfile" "$outfile" || test_fail "test 3 failed" -$RSYNC -aiHvv "$fromdir/" "$todir/" \ - | tee "$outfile" -filter_outfile -cat <<EOT >"$chkfile" +cat >"$chkfile" <<EOT .d..t.$dots ./ cDc.t.$dots block cDc...$dots block2 @@ -157,7 +142,8 @@ if test "$CAN_HLINK_SPECIAL" = no; then grep -v block3.5 <"$chkfile" >"$chkfile.new" mv "$chkfile.new" "$chkfile" fi -diff $diffopt "$chkfile" "$outfile" || test_fail "test 4 failed" + +checkdiff2 "$RSYNC -aiHvv '$fromdir/' '$todir/'" v_filt echo "check how the directory listings compare with diff:" echo "" @@ -167,9 +153,7 @@ diff $diffopt "$tmpdir/ls-from" "$tmpdir/ls-to" if test "$CAN_HLINK_SPECIAL" = yes; then set -x - $RSYNC -aii --link-dest="$todir" "$fromdir/" "$chkdir/" \ - | tee "$outfile" - cat <<EOT >"$chkfile" + checkdiff "$RSYNC -aii --link-dest='$todir' '$fromdir/' '$chkdir/'" <<EOT cd$allspace ./ hD$allspace block hD$allspace block2 @@ -180,7 +164,6 @@ hD$allspace char2 hD$allspace char3 hS$allspace fifo EOT - diff $diffopt "$chkfile" "$outfile" || test_fail "test 5 failed" fi # The script would have aborted on error, so getting here means we've won. diff --git a/testsuite/duplicates.test b/testsuite/duplicates.test index e64b8083..cd2f713a 100644 --- a/testsuite/duplicates.test +++ b/testsuite/duplicates.test @@ -29,8 +29,6 @@ name2="$fromdir/name2" echo "This is the file" > "$name1" ln -s "$name1" "$name2" || test_fail "can't create symlink" -outfile="$scratchdir/rsync.out" - checkit "$RSYNC -avv '$fromdir/' '$fromdir/' '$fromdir/' '$fromdir/' '$fromdir/' '$fromdir/' '$fromdir/' '$fromdir/' '$fromdir/' '$fromdir/' '$todir/'" "$fromdir" "$todir" \ | tee "$outfile" diff --git a/testsuite/exclude.test b/testsuite/exclude.test index fad4fd72..9bc3b542 100644 --- a/testsuite/exclude.test +++ b/testsuite/exclude.test @@ -12,9 +12,6 @@ . "$suitedir/rsync.fns" -chkfile="$scratchdir/rsync.chk" -outfile="$scratchdir/rsync.out" - CVSIGNORE='*.junk' export CVSIGNORE @@ -118,7 +115,8 @@ ln -s too "$fromdir/bar/down/to/foo/sym" # Start to prep an --update test dir mkdir "$scratchdir/up1" "$scratchdir/up2" -touch "$scratchdir/up1/older" "$scratchdir/up2/newer" +touch "$scratchdir/up1/dst-newness" "$scratchdir/up2/src-newness" +touch "$scratchdir/up1/same-newness" "$scratchdir/up2/same-newness" touch "$scratchdir/up1/extra-src" "$scratchdir/up2/extra-dest" # Create chkdir with what we expect to be excluded. @@ -133,7 +131,7 @@ rm "$chkdir"/bar/down/to/foo/to "$chkdir"/bar/down/to/foo/file[235-9] rm "$chkdir"/mid/for/foo/extra # Finish prep for the --update test (run last) -touch "$scratchdir/up1/newer" "$scratchdir/up2/older" +touch "$scratchdir/up1/src-newness" "$scratchdir/up2/dst-newness" # Un-tweak the directory times in our first (weak) exclude test (though # it's a good test of the --existing option). @@ -227,13 +225,13 @@ checkit "$RSYNC -avv $relative_opts --exclude='$fromdir/foo/down' \ '$fromdir/foo' '$todir'" "$chkdir$fromdir/foo" "$todir$fromdir/foo" # Now we'll test the --update option. -$RSYNC -aiO --update touch "$scratchdir/up1/" "$scratchdir/up2/" \ - | tee "$outfile" -cat <<EOT >"$chkfile" +checkdiff "$RSYNC -aiiO --update --info=skip '$scratchdir/up1/' '$scratchdir/up2/'" \ + "grep -v '^\.d$allspace'" <<EOT +dst-newness is newer >f$all_plus extra-src ->f..t.$dots newer +.f$allspace same-newness +>f..t.$dots src-newness EOT -diff $diffopt "$chkfile" "$outfile" || test_fail "--update test failed" # The script would have aborted on error, so getting here means we've won. exit 0 diff --git a/testsuite/hardlinks.test b/testsuite/hardlinks.test index 740d725f..6c4fafbc 100644 --- a/testsuite/hardlinks.test +++ b/testsuite/hardlinks.test @@ -13,8 +13,6 @@ SSH="$scratchdir/src/support/lsh.sh" -outfile="$scratchdir/rsync.out" - # Build some hardlinks fromdir="$scratchdir/from" @@ -70,7 +68,7 @@ echo "This is another file" >"$fromdir/solo" ln "$fromdir/solo" "$chkdir/solo" || test_fail "Can't create hardlink" # Make sure that the checksum data doesn't slide due to an HLINK_BUMP() change. -$RSYNC -aHivc --debug=HLINK5 "$fromdir/" "$chkdir/" | tee "$outfile" +checktee "$RSYNC -aHivc --debug=HLINK5 '$fromdir/' '$chkdir/'" grep solo "$outfile" && test_fail "Erroneous copy of solo file occurred!" # Make sure there's nothing wrong with sending a single file with -H diff --git a/testsuite/itemize.test b/testsuite/itemize.test index 9988cb72..146ed2ed 100644 --- a/testsuite/itemize.test +++ b/testsuite/itemize.test @@ -12,9 +12,6 @@ to2dir="$tmpdir/to2" -chkfile="$scratchdir/rsync.chk" -outfile="$scratchdir/rsync.out" - makepath "$fromdir/foo" makepath "$fromdir/bar/baz" cp_p "$srcdir/configure.ac" "$fromdir/foo/config1" @@ -28,10 +25,18 @@ ln "$fromdir/foo/config1" "$fromdir/foo/extra" rm -f "$to2dir" # Check if rsync is set to hard-link symlinks. -if egrep '^#define CAN_HARDLINK_SYMLINK 1' config.h >/dev/null; then +if $RSYNC --version | grep "[, ] hardlink-symlinks" >/dev/null; then L=hL + sym_dots="$allspace" + L_sym_dots=".L$allspace" + is_uptodate='is uptodate' + touch "$chkfile.extra" else L=cL + sym_dots="c.t.$dots" + L_sym_dots="cL$sym_dots" + is_uptodate='-> ../bar/baz/rsync' + echo "cL$sym_dots foo/sym $is_uptodate" >"$chkfile.extra" fi # Check if rsync can preserve time on symlinks @@ -48,9 +53,7 @@ case "$RSYNC" in ;; esac -$RSYNC -iplr "$fromdir/" "$todir/" \ - | tee "$outfile" -cat <<EOT >"$chkfile" +checkdiff "$RSYNC -iplr '$fromdir/' '$todir/'" <<EOT cd$all_plus ./ cd$all_plus bar/ cd$all_plus bar/baz/ @@ -61,22 +64,18 @@ cd$all_plus foo/ >f$all_plus foo/extra cL$all_plus foo/sym -> ../bar/baz/rsync EOT -diff $diffopt "$chkfile" "$outfile" || test_fail "test 1 failed" # Ensure there are no accidental directory-time problems. $RSYNC -a -f '-! */' "$fromdir/" "$todir" cp_p "$srcdir/configure.ac" "$fromdir/foo/config2" chmod 601 "$fromdir/foo/config2" -$RSYNC -iplrH "$fromdir/" "$todir/" \ - | tee "$outfile" -cat <<EOT >"$chkfile" +checkdiff "$RSYNC -iplrH '$fromdir/' '$todir/'" <<EOT >f..T.$dots bar/baz/rsync >f..T.$dots foo/config1 >f.sTp$dots foo/config2 hf..T.$dots foo/extra => foo/config1 EOT -diff $diffopt "$chkfile" "$outfile" || test_fail "test 2 failed" $RSYNC -a -f '-! */' "$fromdir/" "$todir" cp_p "$srcdir/config.sub" "$fromdir/foo/config2" @@ -88,38 +87,30 @@ umask 022 chmod 600 "$fromdir/foo/config2" chmod 777 "$todir/bar/baz/rsync" -$RSYNC -iplrtc "$fromdir/" "$todir/" \ - | tee "$outfile" -cat <<EOT >"$chkfile" +checkdiff "$RSYNC -iplrtc '$fromdir/' '$todir/'" <<EOT .f..tp$dots bar/baz/rsync .d..t.$dots foo/ .f..t.$dots foo/config1 >fcstp$dots foo/config2 cLc$T.$dots foo/sym -> ../bar/baz/rsync EOT -diff $diffopt "$chkfile" "$outfile" || test_fail "test 3 failed" cp_p "$srcdir/configure.ac" "$fromdir/foo/config2" chmod 600 "$fromdir/foo/config2" # Lack of -t is for unchanged hard-link stress-test! -$RSYNC -vvplrH "$fromdir/" "$todir/" \ - | tee "$outfile" -filter_outfile -cat <<EOT >"$chkfile" +checkdiff "$RSYNC -vvplrH '$fromdir/' '$todir/'" \ + v_filt <<EOT bar/baz/rsync is uptodate foo/config1 is uptodate foo/extra is uptodate foo/sym is uptodate foo/config2 EOT -diff $diffopt "$chkfile" "$outfile" || test_fail "test 4 failed" chmod 747 "$todir/bar/baz/rsync" $RSYNC -a -f '-! */' "$fromdir/" "$todir" -$RSYNC -ivvplrtH "$fromdir/" "$todir/" \ - | tee "$outfile" -filter_outfile -cat <<EOT >"$chkfile" +checkdiff "$RSYNC -ivvplrtH '$fromdir/' '$todir/'" \ + v_filt <<EOT .d$allspace ./ .d$allspace bar/ .d$allspace bar/baz/ @@ -130,47 +121,23 @@ cat <<EOT >"$chkfile" hf$allspace foo/extra .L$allspace foo/sym -> ../bar/baz/rsync EOT -diff $diffopt "$chkfile" "$outfile" || test_fail "test 5 failed" chmod 757 "$todir/foo/config1" touch "$todir/foo/config2" -$RSYNC -vplrtH "$fromdir/" "$todir/" \ - | tee "$outfile" -filter_outfile -cat <<EOT >"$chkfile" +checkdiff "$RSYNC -vplrtH '$fromdir/' '$todir/'" \ + v_filt <<EOT foo/config2 EOT -diff $diffopt "$chkfile" "$outfile" || test_fail "test 6 failed" chmod 757 "$todir/foo/config1" touch "$todir/foo/config2" -$RSYNC -iplrtH "$fromdir/" "$todir/" \ - | tee "$outfile" -cat <<EOT >"$chkfile" +checkdiff "$RSYNC -iplrtH '$fromdir/' '$todir/'" <<EOT .f...p$dots foo/config1 >f..t.$dots foo/config2 EOT -diff $diffopt "$chkfile" "$outfile" || test_fail "test 7 failed" -$RSYNC -ivvplrtH --copy-dest=../to "$fromdir/" "$to2dir/" \ - | tee "$outfile" -filter_outfile -- The rsync repository. _______________________________________________ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs