Re: [Bug 11521] rsync does not use high-resolution timestamps to determine file differences

[f-rsync]

> Date: Sun, 24 Jan 2016 15:43:20 -0800
> From: Wayne Davison <wa...@thedavisons.net>

A couple questions below; please bear with me.

> No, if you do a ext4 -> ext4 copy, rsync has set the matching ns info for
> transferred files since 3.1.0. There was a case prior to rsync 3.1.2 where
> a brand-new file transferred in the same second it was created wouldn't 
get
> the right ns value because rsync was optimizing away the time-set if the
> file's mod-time matched in the integer part (3.1.2 fixed that).

Oh, I see what happened.  My problem is that no Ubuntu LTS before
14.04 had rsync 3.1.0 or newer, and the original capability took more
than four years to make it into a released rsync version, if I'm
reading the release notes correctly.*  Unfortunately, that means
the vast majority of my machine base predates the fix, including the
machine hosting the backups.  I can obviously install newer rsyncs,
but that gives me a big installed base of pre-fix data that I'm going
to have to fix, and no more rsync security updates unless I track them
manually.  Yet I'd rather do this now, so I'm future-proofed, than
be badly surprised some years down the road when this rsync behavior
becomes the default, and to keep the problem from continuing to get worse.

* I may have tried 3.1.0 at some point and then realized the problems
  it'd give me for backups and didn't install it everywhere, pending
  a better fix; this is starting to ring bells.  I'm really surprised
  that the initial patch of Sep 7, 2009 never made it into a released
  rsync until 3.1.0 of Sep 28 2013; that's a four-year delay, and
  explains why I obviously never tried this out until perhaps an
  experiment with 3.1.0, and no doubt I didn't want to run a private
  version that wouldn't get security updates.  Ubuntu rsync versions:
  10.04 has 3.0.7 proto 30; 12.04 has 3.0.9 p 30; 14.04 has 3.1.0 p 31.)

So what it looks like is that the capability to transfer ns times at
all existed in CVS but not released since 2009, in released since
2013, and in an Ubuntu LTS since 2014.  And the current patch -seems-
to be an optimization that avoids -comparison- if the ns times match,
but that only affects speed---it doesn't change what gets written in
any event, just how fast.  Right?  But actually I think it changes
behavior besides that---see my test case below.

> Beginning with this patch you can run rsync -aiv --checksum -@-1 and have
> it fix the full mod-time on any matching files it finds. But for most 
newer
> backups, the ns time will already be set correctly (as long as it was
> created using a new enough rsync and protocol 31). If someone has a large
> link-dest hierarchy that predates 3.1.0, then you could be sharing
> hard-linked matching files from back before the ns info was included (the
> older files would all have 0 for the ns value).

Wow, that -c really hurts.  If one wanted to live dangerously---with
the assumption that two files that otherwise match in all metadata
(including obviously length :) but whose timestamps differ in that one
has integer seconds and the other has the same integer seconds but
also nanoseconds, can rsync readjust the dates, without doing a full
checksum?  If not, I may write such a tool, or do it the (very) slow
way and have rsync re-checksum a few terabytes of my backups... :)
[Might find some bitrot that way, of course.]

Also, I actually -can't- use that command to fix my snapshots,
because (if I understand correctly), it will -alter- my existing
snapshots to match the -current- contents of files, destroying them
---I'll no longer be able to go back in time to a previous version.
I only want to update ns times on files in the older snapshots
if and only if changing integer times to ns times would be the
only modification.  I think rsync -ac -@-1 will do far more, yes?

As for -@-1, that introduces a surprising change in behavior when I
try it.  I'm unsure if it's intended, though I think it is.  But it
will -definitely- break my hardlinks and bloat the backups if I try
it without readjusting the dates in --link-dest directories (e.g.,
previous snapshots).  I find that specifying -@-1 copies the ns
timestamp from the source to the destination even if the --link-dest
directory has an integer timestamp, and so I assume this is part of
the purpose of the patch?  Not just an optimization, but a change in
the way --link-dest might work.  Observe:

22:57:22 ~$ mkdir T
22:57:25 ~$ cd T
22:57:26 ~/T$ mkdir 1 2 3 4 5 6
22:57:30 ~/T$ lat() { ls -alF -i --full-time "$@"; }
22:57:49 ~/T$ touch 1/foo
22:57:53 ~/T$ ln 1/foo 1/bar
22:57:56 ~/T$ lat */*
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/bar
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/foo
22:57:59 ~/T$ rsync -aviH 1/ 2/
sending incremental file list
.d..t.. ./
>f+ foo
hf+ bar => 

[Bug 11521] rsync does not use high-resolution timestamps to determine file differences

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=11521

Wayne Davison  changed:

   What|Removed |Added

 Status|NEW |RESOLVED
 Resolution|--- |FIXED

--- Comment #3 from Wayne Davison  ---
The latest git version has an option that lets you choose to include
nanoseconds in comparisons if you want them. Having it on by default would
likely cause far too many headaches for various backup solutions that use an
older filesystem (e.g. ext3) that doesn't support nanoseconds.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 11521] rsync does not use high-resolution timestamps to determine file differences

[f-rsync]

[Included text copied from the commit.]

> Date: Sun, 24 Jan 2016 19:52:49 +
> From: samba-b...@samba.org

> Auto-Submitted: auto-generated

> https://bugzilla.samba.org/show_bug.cgi?id=11521

> Wayne Davison <way...@samba.org> changed:

>What|Removed |Added
> 

>  Status|NEW |RESOLVED
>  Resolution|--- |FIXED

> --- Comment #3 from Wayne Davison <way...@samba.org> ---
> The latest git version has an option that lets you choose to include
> nanoseconds in comparisons if you want them. Having it on by default would
> likely cause far too many headaches for various backup solutions that use 
an
> older filesystem (e.g. ext3) that doesn't support nanoseconds.

Thanks for the patch!

Just FYI, this comment is true but incomplete---the scenario I was
describing was straight ext4-to-ext4 copies and/or backups.  The
timestamp problem I currently see with those is because rsync was
throwing away the nanosecond information until this patch, even though
both ends supported it.

Anyone who's used dirvish, or presumably similar tools such as
rsnapshot, from and to ext4 or other ns-supporting filesystems, will
be bitten by the problem of non-ns vs ns timestamps bloating backups
and breaking hardlinks, either when they manually use --modify-window=-1,
or when this becomes the default.

I'd still love to see either some in-rsync workaround that can be left
in place*, or (second best) some clever by-hand one-time workaround
that uses rsync just once to update all those timestamps while not
breaking the hardlinks (presumably -not- using --link-dest for those),
or (third best) some not-rsync-at-all solution that basically does
a giant ls at the source and a giant touch at the destination.  Yes,
I know that none of those can fix up backed-up files that are no
longer in the same place in the source; those at least won't bloat/
unhardlink later backups.

This solution is likely going to have to be reinvented repeatedly by
people running backups, so working out the right way to do it and then
putting it somewhere other rsync users will find it is likely a useful
exercise.  Before I solve it by hand for myself, it'd be useful to
know if it's likely rsync will come up with some way to solve it for
me.

* E.g., (a) "If modify-window is -1, and --link-dest is in use, and
everything else about the file or directory matches -except- the
timestamp, then update the timestamp," and (b) "Provide a switch to
turn off this heuristic after I'm sure my backups are okay."  (The
intent of (b) is to catch later slight changes of timestamp but
actually record them as -separate files-, which is important so
older snapshots don't magically change out from under you if something
updates a timestamp by a fraction of a second without otherwise
changing anything, which does happen and can occasionally be -very-
important to know about when tracking down issues.)

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 11521] New: rsync does not use high-resolution timestamps to determine file differences

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=11521

Bug ID: 11521
   Summary: rsync does not use high-resolution timestamps to
determine file differences
   Product: rsync
   Version: 3.1.2
  Hardware: All
OS: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
  Assignee: way...@samba.org
  Reporter: michael.mccrac...@gmail.com
QA Contact: rsync...@samba.org

The sub-second timestamps available on many filesystems are preserved when
requested across copies, but aren't used to determine file differences.

If a file exists at both origin and destination and its contents the same size
in each place, and the timestamps only differ in the sub-second resolution,
rsync will treat the files as the same (unless you use --checksum).

So if a file is created, and then a snapshot of its dir is taken, then the
origin file is modified (but the size is preserved) within the same second, an
attempt to update that snapshot using rsync will fail to copy the change.

Here's a script that reproduces the issue with high reliability for me:

#!/bin/bash 

set -x

DIR=$(mktemp -d -p $(pwd))

mkdir $DIR/d1
mkdir $DIR/d2

echo dummy > $DIR/d1/dummy
echo dummy > $DIR/d2/dummy

echo one > $DIR/d1/afile
sleep 0.1
echo two > $DIR/d2/afile

/usr/bin/stat $DIR/d1/afile | grep Mod
/usr/bin/stat $DIR/d2/afile | grep Mod

~/packages/rsync/rsync --delete -a -HAX -vii $DIR/d2/ $DIR/d1

diff -r $DIR/d1 $DIR/d2

/usr/bin/stat $DIR/d1/afile | grep Mod
/usr/bin/stat $DIR/d2/afile | grep Mod



If the diff shows a difference, then the rsync didn't copy afile's contents
over. However, note the stat info from the last two lines - the updated modify
timestamp *will* be synced, making an inconsistent sync.

The following patch adds a check of the high-res timestamp to unchanged_file.
This solves the problem for me, and I've guarded it so it shouldn't break on
systems with no high-res timestamp. Please let me know if I can be helpful in
testing it further or making it more robust.



diff --git a/generator.c b/generator.c
index 3a4504f..2f64f5d 100644
--- a/generator.c
+++ b/generator.c
@@ -588,7 +588,11 @@ int unchanged_file(char *fn, struct file_struct *file,
STRUCT_STAT *st)
if (ignore_times)
return 0;

-   return cmp_time(st->st_mtime, file->modtime) == 0;
+   return cmp_time(st->st_mtime, file->modtime) == 0
+#ifdef ST_MTIME_NSEC
+   && st->ST_MTIME_NSEC == F_MOD_NSEC(file)
+#endif
+   ;
 }

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 11521] rsync does not use high-resolution timestamps to determine file differences

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=11521

--- Comment #1 from Michael McCracken  ---
Created attachment 11440
  --> https://bugzilla.samba.org/attachment.cgi?id=11440=edit
patch to check hi-res timestamp in unchanged_file

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: The easiest way to restore timestamps of files?

[Frantisek Hanzlik]

Kevin Korb wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 - --archive --itemize-changes --size-only --dry-run
 Obviously the last one is so you can confirm what it is going to do.
 
 On 04/26/2015 06:08 PM, Frantisek Hanzlik wrote:
 Please, is possible (with rsync) re-create files timestamps? I mean
 something like choosing a -T in the program mirror- from it's man
 page e.g. there: 
 http://sunsite.univie.ac.at/textbooks/mirror/mirror.html#Flags

 Do not do any file transfers just force the time-stamps of any
 local files to be reset to be the same as the remote files.
 Normally only used when initialising a mirror that already contains
 files retrieved another way (e.g. from CDROM).

 This is the situation I have encountered several times - I have
 local files which was downloaded from internet archive via FTP or
 obtained somehow, these files have bad timestamp, and now I want
 use rsync for their synchronization with the remote rsync server
 and also correct their bad timestamp - of course without
 re-downloading. Is it possible?

Hi Kevin,
thanks for your help, it works. My mistake was that I thought that
the use of the '--size-only' switch will not repair file timestamps.

Now I still do not understand why does not work this switch form:

--chmod=D=775,F=664

(it causes error message:
rsync: Invalid argument passed to --chmod (D=775,F=664)
rsync error: syntax or usage error (code 1) at main.c(1554) [client=3.1.0]
). But it isn't too big problem, as form:

--chmod=Dug=rwx,Do=rx,Fug=rw,Fo=r

works well. It seems as compatibility with chmod syntax isn't perfect
and rsync man page not clearly mentioned this.

Thanks again, Franta Hanzlik

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: The easiest way to restore timestamps of files?

[Francis . Montagnac]

Hi
On Wed, 29 Apr 2015 14:23:56 +0200 Frantisek Hanzlik wrote:

 Now I still do not understand why does not work this switch form:

 --chmod=D=775,F=664

This is not the proper syntax: suppress the = signs before the modes:

--chmod=D775,F664

--
Francis
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

The easiest way to restore timestamps of files?

[Frantisek Hanzlik]

Please, is possible (with rsync) re-create files timestamps?
I mean something like choosing a -T in the program mirror- from
it's man page e.g. there:
http://sunsite.univie.ac.at/textbooks/mirror/mirror.html#Flags

Do not do any file transfers just force the time-stamps of any local
files to be reset to be the same as the remote files. Normally only
used when initialising a mirror that already contains files retrieved
another way (e.g. from CDROM).

This is the situation I have encountered several times - I have local
files which was downloaded from internet archive via FTP or obtained
somehow, these files have bad timestamp, and now I want use rsync for
their synchronization with the remote rsync server and also correct
their bad timestamp - of course without re-downloading. Is it possible?
-- 
Thanks in advance, Franta Hanzlik

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: The easiest way to restore timestamps of files?

[Kevin Korb]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

- --archive --itemize-changes --size-only --dry-run
Obviously the last one is so you can confirm what it is going to do.

On 04/26/2015 06:08 PM, Frantisek Hanzlik wrote:
 Please, is possible (with rsync) re-create files timestamps? I mean
 something like choosing a -T in the program mirror- from it's man
 page e.g. there: 
 http://sunsite.univie.ac.at/textbooks/mirror/mirror.html#Flags
 
 Do not do any file transfers just force the time-stamps of any
 local files to be reset to be the same as the remote files.
 Normally only used when initialising a mirror that already contains
 files retrieved another way (e.g. from CDROM).
 
 This is the situation I have encountered several times - I have
 local files which was downloaded from internet archive via FTP or
 obtained somehow, these files have bad timestamp, and now I want
 use rsync for their synchronization with the remote rsync server
 and also correct their bad timestamp - of course without
 re-downloading. Is it possible?
 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iEYEARECAAYFAlU9bp8ACgkQVKC1jlbQAQcgBwCg4I0VpJfxWYWCBiLJo2qKMWpj
6ooAoJIqtmC6juv7KmqHSCWeUtRielEE
=mumk
-END PGP SIGNATURE-
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 8198] rsync does not set TZ variable after chroot(), which confuses logging timestamps

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=8198

Wayne Davison way...@samba.org changed:

   What|Removed |Added

 Status|NEW |RESOLVED
 Resolution||WONTFIX

--- Comment #4 from Wayne Davison way...@samba.org 2011-06-04 16:37:15 UTC ---
I do think that if the chroot call doesn't find a new timezone setup, that it
should leave things alone.  That is how things have always worked before, and
supports minimal chroot areas better.  So, fundamentally, I think this is a
glibc issue.

A user should be able to work around this by setting their own TZ string before
starting up the daemon.  For instance, I have sometimes used this TZ for other
purposes:

export TZ=PST8PDT

Note that this setting fully supports daylight savings time changes.  If rsync
were tweaked to set a TZ var, it would be nice if it could construct such an
all-encompassing export, but that does seem rather complex to try to code up,
and is probably more trouble than it's worth.

So, I'm not particularly liking the TZ idea, especially since a user affected
by a buggy libc has an easy way to configure this externally to rsync.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 8198] rsync does not set TZ variable after chroot(), which confuses logging timestamps

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=8198

--- Comment #5 from Matt McCutchen m...@mattmccutchen.net 2011-06-04 18:04:35 
UTC ---
(In reply to comment #4)
 I do think that if the chroot call doesn't find a new timezone setup, that it
 should leave things alone.  That is how things have always worked before, and
 supports minimal chroot areas better.

The chroot call doesn't do anything with timezones.  The functions that make
use of the timezone are periodically checking /etc/localtime to respond to
system configuration changes.

What you suggest fails if the in-chroot module path is empty and there happens
to be an /etc/localtime file in the module, or if another application chroots
into a POSIX-like system that omits /etc/localtime with the intent that the
default be used.  I maintain that the right solution is for the application to
tell glibc whether to freeze the configuration upon chroot.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 8198] rsync does not set TZ variable after chroot(), which confuses logging timestamps

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=8198

--- Comment #2 from Matt McCutchen m...@mattmccutchen.net 2011-06-04 03:25:27 
UTC ---
The solution is a little messy and fails in the rare event that an rsync run
straddles a DST change.

Fundamentally, glibc is assuming that the new root directory is that of a
POSIX-like system whose configuration it should observe, which is not the case.
 If what rsync is doing is considered a valid use case for chroot, there should
be a way to advise glibc so it does not try to reload configuration.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 8198] rsync does not set TZ variable after chroot(), which confuses logging timestamps

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=8198

--- Comment #3 from Matt McCutchen m...@mattmccutchen.net 2011-06-04 03:36:12 
UTC ---
E.g., the application could call a function freeze_config(int items) to preload
the configuration items it will need as indicated by the items bitmask and
then suppress future loading of configuration.  For diagnostic purposes, glibc
functions could fail if they use configuration items whose necessity was not
declared to freeze_config, even if the application was lucky enough to have
loaded the items for another reason.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 8198] New: rsync does not set TZ variable after chroot(), which confuses logging timestamps

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=8198

   Summary: rsync does not set TZ variable after chroot(), which
confuses logging timestamps
   Product: rsync
   Version: 3.1.0
  Platform: All
OS/Version: All
Status: NEW
  Severity: normal
  Priority: P5
 Component: core
AssignedTo: way...@samba.org
ReportedBy: jkal...@redhat.com
 QAContact: rsync...@samba.org


Hi,
after chroot(), current timezone is changed in chrooted process which leads
to the following output in logs:

May 29 20:00:35 nec-em12 rsyncd[9770]: connect from localhost.localdomain
(127.0.0.1) 
May 29 20:00:35 nec-em12 rsyncd[9770]: rsync to my_data/ from
localhost.localdomain (127.0.0.1) 
May 30 00:00:35 nec-em12 rsyncd[9770]: receiving file list 
May 30 00:00:35 nec-em12 rsyncd[9770]: client/ 
May 30 00:00:35 nec-em12 rsyncd[9770]: client/data/

Proper time is 20:00:35.

Attached patch fixes this issue by caching current timezone and setting TZ
variable after chroot() call.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 8198] rsync does not set TZ variable after chroot(), which confuses logging timestamps

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=8198

--- Comment #1 from Jan Kaluza jkal...@redhat.com 2011-06-01 10:07:01 UTC ---
Created attachment 6508
  -- https://bugzilla.samba.org/attachment.cgi?id=6508
patch

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [Bug 8198] New: rsync does not set TZ variable after chroot(), which confuses logging timestamps

[Michael Tokarev]

01.06.2011 14:06, samba-b...@samba.org пишет:
 https://bugzilla.samba.org/show_bug.cgi?id=8198
 
Summary: rsync does not set TZ variable after chroot(), which
 confuses logging timestamps
Product: rsync
Version: 3.1.0
   Platform: All
 OS/Version: All
 Status: NEW
   Severity: normal
   Priority: P5
  Component: core
 AssignedTo: way...@samba.org
 ReportedBy: jkal...@redhat.com
  QAContact: rsync...@samba.org
 
 
 Hi,
 after chroot(), current timezone is changed in chrooted process which leads
 to the following output in logs:

 Attached patch fixes this issue by caching current timezone and setting TZ
 variable after chroot() call.

tzset() call before chroot should be enough for that --
that's what has always been done in chroot programs
exactly for this purpose (tzset was broken in glibc
at least once, but that's details).

/mjt

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Backups Directory Timestamps Not Preserved

[Michal Soltys]

Backup options are concered with files from what I can see after some 
tests. Extra directory (--backup-dir) is an addition to keep things more 
tidy. I guess the idea was to have some sort of protection against 
careless rsync invocation, etc. - not as a solution for incremental 
backups. I don't really know for sure if it's intended or not though.


As for incremental backup, --link-dest is generally used for that 
purpose, and saves a lot of space due to hardlink usage. You will likely 
need to make some shell script for automation's sake (or you can look 
for premade ones - e.g. rsnapshot is often quoted).

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Backups Directory Timestamps Not Preserved

[Robert Boucher]

Hello,

I've been testing out using rsync for nightly incremental backups
through the '--backup'  '--backup-dir' options. So far, I've noticed
two issues.

1) First, if an empty directory is removed from the source, rsync will
remove it from the destination but not create the directory in the
backup-dir location.

 $ mkdir -p /tmp/src/foo
 $ rsync -a /tmp/src/ /tmp/dest/
 $ rmdir /tmp/src/foo
 $ rsync -a --delete -P -b --backup-dir=/tmp/desti/ /tmp/src/ /tmp/dest/

2) Second, when the contents of a non-empty directory is modified, the
original timestamp is not preserved in the backup-dir location.

 $ mkdir -p /tmp/src/foo
 $ touch -t 200904011200 /tmp/src/foo/bar /tmp/src/foo /tmp/src
 $ rsync -a /tmp/src/ /tmp/dest/
 $ rm -rf /tmp/src/foo/bar
 $ rsync -a --delete -P -b --backup-dir=/tmp/desti/ /tmp/src/ /tmp/dest/
 $ ls -ald /tmp/desti/foo /tmp/desti/foo/bar

I'm using version 3.0.6. I've played around with various options but did
not find a combination where this worked. I scanned through the source,
and it appears as though rsync only creates directories as required by
files when they are moved over and doesn't update the timestamps after
the directory's contents are processed. Is this accurate or is there
some combination which will allow this to work?

-Rob

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Conflicting timestamps in rsyncd.log

[Bill Landry]

Hi list,

I just setup my first rsyncd server and all is working well as far as
file syncing goes.  However, I am a bit baffled by the fact that some of
the log entries in the rsyncd.log file are entered in local time and
other are entered in GMT.  Here is a sample:

file upload:

2009/03/01 15:15:49 [22641] connect from UNKNOWN (xxx.xxx.xxx.xxx)
2009/03/01 23:15:49 [22641] rsync to some-module/ from some-n...@unknown
(xxx.xxx.xxx.xxx)
2009/03/01 23:15:49 [22641] receiving file list
2009/03/01 23:15:49 [22641] unknown xxx.xxx.xxx.xxx some-module
(some-name) some-file.txt
2009/03/01 23:15:49 [22641] sent 168 bytes  received 6258 bytes  total
size 37421


file download:
==
2009/03/01 15:21:06 [22691] connect from UNKNOWN (xxx.xxx.xxx.xxx)
2009/03/01 23:21:06 [22691] rsync on some-module/some-file.txt from
unknown (xxx.xxx.xxx.xxx)
2009/03/01 23:21:06 [22691] building file list
2009/03/01 23:21:06 [22691] unknown xxx.xxx.xxx.xxx some-module ()
some-file.txt
2009/03/01 23:21:06 [22691] sent 6299 bytes  received 54 bytes  total
size 37421
==

Note the different (but correct local time) timestamps for connect and
the others (which are GMT).  I would like to log all entries with my
local time rather than GMT.  Here is what I have setup for logging in my
rsyncd.conf:

log file = /var/log/rsyncd.log
log format = %h %a %m (%u) %f

Is there any way to resolve this?  Any suggestions would be greatly
appreciated!

Best regards,

Bill
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Conflicting timestamps in rsyncd.log

[Matt McCutchen]

On Sun, 2009-03-01 at 17:21 -0800, Bill Landry wrote:
 I just setup my first rsyncd server and all is working well as far as
 file syncing goes.  However, I am a bit baffled by the fact that some of
 the log entries in the rsyncd.log file are entered in local time and
 other are entered in GMT.

This happens because the chroot done by the rsync daemon prevents glibc
from reading /etc/localtime to determine the current timezone.  I'm not
sure what is the right fix for this issue, but you can work around it
using the new chroot+chdir mode in rsync = 3.0.0.  E.g., configuration:

[some-module]
use chroot = true
path = /path/to/chroot/./some-module

Then copy /etc/localtime to /path/to/chroot/etc/localtime and put the
module data in /path/to/chroot/some-module.  See also:

https://bugzilla.samba.org/show_bug.cgi?id=4402

-- 
Matt

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Conflicting timestamps in rsyncd.log

[Eberhard Moenkeberg]

Hi,

On Sun, 1 Mar 2009, Matt McCutchen wrote:
 On Sun, 2009-03-01 at 17:21 -0800, Bill Landry wrote:

  I just setup my first rsyncd server and all is working well as far as
  file syncing goes.  However, I am a bit baffled by the fact that some of
  the log entries in the rsyncd.log file are entered in local time and
  other are entered in GMT.
 
 This happens because the chroot done by the rsync daemon prevents glibc
 from reading /etc/localtime to determine the current timezone.  I'm not
 sure what is the right fix for this issue, but you can work around it
 using the new chroot+chdir mode in rsync = 3.0.0.  E.g., configuration:
 
 [some-module]
   use chroot = true
   path = /path/to/chroot/./some-module
 
 Then copy /etc/localtime to /path/to/chroot/etc/localtime and put the
 module data in /path/to/chroot/some-module.  See also:
 
 https://bugzilla.samba.org/show_bug.cgi?id=4402

A nice workaround, but not a solution.
The daemon should get and export the timezone info before forking into 
the chroot environment.


Viele Grüße
Eberhard Mönkeberg (emoe...@gwdg.de, e...@kki.org)

-- 
Eberhard Mönkeberg
Arbeitsgruppe IT-Infrastruktur
E-Mail: emoe...@gwdg.de  Tel.: +49 (0)551 201-1551
-
Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG)
Am Fassberg 11, 37077 Göttingen
URL:http://www.gwdg.de E-Mail: g...@gwdg.de
Tel.:   +49 (0)551 201-1510Fax:+49 (0)551 201-2150
Geschäftsführer:   Prof. Dr. Bernhard Neumair
Aufsichtsratsvorsitzender: Dipl.-Kfm. Markus Hoppe
Sitz der Gesellschaft: Göttingen
Registergericht:   Göttingen  Handelsregister-Nr. B 598
--- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Conflicting timestamps in rsyncd.log

[Bill Landry]

Eberhard Moenkeberg wrote:
 Hi,
 
 On Sun, 1 Mar 2009, Matt McCutchen wrote:
 On Sun, 2009-03-01 at 17:21 -0800, Bill Landry wrote:
 
 I just setup my first rsyncd server and all is working well as far as
 file syncing goes.  However, I am a bit baffled by the fact that some of
 the log entries in the rsyncd.log file are entered in local time and
 other are entered in GMT.
 This happens because the chroot done by the rsync daemon prevents glibc
 from reading /etc/localtime to determine the current timezone.  I'm not
 sure what is the right fix for this issue, but you can work around it
 using the new chroot+chdir mode in rsync = 3.0.0.  E.g., configuration:

 [some-module]
  use chroot = true
  path = /path/to/chroot/./some-module

 Then copy /etc/localtime to /path/to/chroot/etc/localtime and put the
 module data in /path/to/chroot/some-module.  See also:

 https://bugzilla.samba.org/show_bug.cgi?id=4402
 
 A nice workaround, but not a solution.
 The daemon should get and export the timezone info before forking into 
 the chroot environment.

Thanks for the reply, I could copy localtime into the chroot.  I just
tried using a logging facility instead of a log file by changing the
logging in my rsyncd.conf to:

log format = %h %a %m (%u) %f
syslog facility = local3

and adding the following to my rsyslog.conf file:

local3.info/var/log/rsyncd.log

However, now the rsync logging is going into both /var/log/rsyncd.log
and /var/log/messages.  If it's not one thing it's another.  Is there a
way to only log rsync messages to /var/log/rsyncd.log and not to
/var/log/messages?

Thanks for any feedback!

Bill
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Conflicting timestamps in rsyncd.log

[Bill Landry]

Matt McCutchen wrote:
 On Sun, 2009-03-01 at 17:21 -0800, Bill Landry wrote:
 I just setup my first rsyncd server and all is working well as far as
 file syncing goes.  However, I am a bit baffled by the fact that some of
 the log entries in the rsyncd.log file are entered in local time and
 other are entered in GMT.
 
 This happens because the chroot done by the rsync daemon prevents glibc
 from reading /etc/localtime to determine the current timezone.  I'm not
 sure what is the right fix for this issue, but you can work around it
 using the new chroot+chdir mode in rsync = 3.0.0.  E.g., configuration:
 
 [some-module]
   use chroot = true
   path = /path/to/chroot/./some-module
 
 Then copy /etc/localtime to /path/to/chroot/etc/localtime and put the
 module data in /path/to/chroot/some-module.  See also:
 
 https://bugzilla.samba.org/show_bug.cgi?id=4402

Thanks for this workaround/solution, Matt.  At least this works.

Bill
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Problem handling future timestamps?

[Wayne Davison]

On Thu, Apr 19, 2007 at 07:26:40PM -0700, Carson Gaspar wrote:
 Emit a WARNING and set to MAX_TIME_T on the destination? It's either
 that or an ERROR and fail the transfer.

I had considered forcing the time to MAX_TIME_T, but decided against
it because I didn't want all too-large timestamps to end up with an
identical time value (since it would make finding changed files
difficult).  I ended up outputting a warning and allowing the value
to get truncated.  This preserves the ability for rsync to figure
out if a file has changed, though it does end up setting the file's
time into the past.

..wayne..
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Problem handling future timestamps?

[Jon Burgess]

I use rsync as a backup tool (via rsnapshot) and noticed that it had a
problem with a couple of files which had timestamps way off in the
future. You can reproduce the problem quite simply:

$ touch -t 207608011200 foo
$ rsync -a foo bar
$ ls -l foo bar
-rw-rw-r-- 1 jburgess jburgess 0 Jun 26  1940 bar
-rw-rw-r-- 1 jburgess jburgess 0 Aug  1  2076 foo

Since the timestamp appears different between the original and the
backup, I get a new copy of the file in every backup.

This was with rsync-2.6.9 on a FC6 x86_64 machine running linux-2.6.20
on an ext3 filesystem. I see the same on a freshly compiled 3.0.0cvs
version too.

Jon


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Problem handling future timestamps?

[Wayne Davison]

On Thu, Apr 19, 2007 at 05:20:59PM +0100, Jon Burgess wrote:
 I use rsync as a backup tool (via rsnapshot) and noticed that it had a
 problem with a couple of files which had timestamps way off in the
 future.

That's a unix-time limitation.  The current timestamp resolution can't
represent anything past January 18, 2038.  Fixing that won't be simple,
but should not really be needed for quite a few more years.

..wayne..
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Problem handling future timestamps?

[Jon Burgess]

On Thu, 2007-04-19 at 10:30 -0700, Wayne Davison wrote:
 On Thu, Apr 19, 2007 at 05:20:59PM +0100, Jon Burgess wrote:
  I use rsync as a backup tool (via rsnapshot) and noticed that it had a
  problem with a couple of files which had timestamps way off in the
  future.
 
 That's a unix-time limitation.  The current timestamp resolution can't
 represent anything past January 18, 2038.  Fixing that won't be simple,
 but should not really be needed for quite a few more years.
 
 ..wayne..

I agree in part -- but the fact the the file exists on a current
filesystem goes to show that it is a problem which exists today. Other
tools like touch and ls clearly handle these timestamps with no problem.

I can obviously work around the problem by resetting the bad timestamps,
but I thought i'd raise it as an issue since it may be causing to rsync
to scan files unnecessarily in some cases.

Files do occasionally get broken timestamps for all kinds of reasons.

Jon


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Problem handling future timestamps?

[Wayne Davison]

On Thu, Apr 19, 2007 at 07:19:00PM +0100, Jon Burgess wrote:
 The particular problem I see where the timestamps 1940 != 2076 is due to
 a 64 bit time_t. On such platforms, increasing the modtime protocol
 entity from 4 to 8 bytes is sufficient to fix this.

Good point.  I'm going to include the ability to transfer a  4-byte
time_t value for protocol 30.  We'll need to consider what should be
done if a value is too large for the destination system to handle.

..wayne..
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Problem handling future timestamps?

[Carson Gaspar]

Wayne Davison wrote:

On Thu, Apr 19, 2007 at 05:20:59PM +0100, Jon Burgess wrote:

I use rsync as a backup tool (via rsnapshot) and noticed that it had a
problem with a couple of files which had timestamps way off in the
future.


That's a unix-time limitation.  The current timestamp resolution can't
represent anything past January 18, 2038.  Fixing that won't be simple,
but should not really be needed for quite a few more years.


Which is what they said about y2k. People never learn. *sigh*

(To clarify, as my use of the word troll in my last message was 
misunderstood, I assume by non-USENET users, I am referring to the utter 
lack of OS support. Obviously rsync can't do very much until the OS 
vendors get their acts together.)


--
Carson
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Problem handling future timestamps?

[Carson Gaspar]

Wayne Davison wrote:


Good point.  I'm going to include the ability to transfer a  4-byte
time_t value for protocol 30.  We'll need to consider what should be
done if a value is too large for the destination system to handle.


Emit a WARNING and set to MAX_TIME_T on the destination? It's either 
that or an ERROR and fail the transfer.


--
Carson
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

DO NOT REPLY [Bug 3479] New: Request: timestamps in --log-format

[bugzilla-daemon]

https://bugzilla.samba.org/show_bug.cgi?id=3479

   Summary: Request: timestamps in --log-format
   Product: rsync
   Version: 2.6.7
  Platform: Other
OS/Version: Linux
Status: NEW
  Severity: normal
  Priority: P3
 Component: core
AssignedTo: [EMAIL PROTECTED]
ReportedBy: [EMAIL PROTECTED]
 QAContact: [EMAIL PROTECTED]


It will be very helpful to have the ability to include information about the
modification time of the files. This will help in (1) creating customized
update file list (for --files-from) (2) for logging. Something like '%M'. Any
datetime format will work (either the Unix, or the standard MMDDTHHMMSS).

The problem that I'm trying to address is the maintainance of a backup site in
a high-traffic environment. The update should handle high priority files first
(priority is based on file age (recent changes first), file size (small changes
first), and file name/type (for example: PDF files last).

If the information about the source files is available, one can use rsync -n
to get the list of files, implement a custom sort/filter, and feed the list to
rsync --from-file session.

Extra points for showing the comparable time on the recieving side (if the file
exists), and for option to show the file owner (id/name)


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug, or are watching the QA contact.
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

DO NOT REPLY [Bug 3479] Request: timestamps in --log-format

[bugzilla-daemon]

https://bugzilla.samba.org/show_bug.cgi?id=3479


[EMAIL PROTECTED] changed:

   What|Removed |Added

   Severity|normal  |enhancement
 Status|NEW |RESOLVED
 Resolution||FIXED




--- Comment #1 from [EMAIL PROTECTED]  2006-02-03 15:39 MST ---
Makes sense to me.  Since we already allow the user to output the length, why
not the modtime, uid, gid, and the mode-bits too while we're at it.  (I didn't
go for the extra points, since outputting receiver-side attributes would be a
lot more complicated to code up.)

I've checked-in a change to the CVS version.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug, or are watching the QA contact.
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync.log - two differ timestamps

[Steve Magee]

Just upgraded from rsync 2.6.2  protocol version 28 to 2.6.3 protocol 
version 28.

Linux kernel upgraded from 2.6.5-1 to 2.6.9-11.

From the client side, I issued...
rsync -av web::app_web /app   /dev/null

From the server side, the config shows...
[app_web]
 comment = Moving webs Inside  Outside
 path = /app
 auth users = root
 secrets file = /etc/rsyncd.secrets
 read only = true
 uid = root

In /var/log/rsync.log, I get two records recorded for the single issue 
or rsync from the
client side.  The problem is the timestamp are using localtime and 
Greenwich time.  While
running on the older version of the kernel, both timestamps were using 
localtime.


Jul 13 16:44:59 web rsyncd[26323]: rsync on app_web from 
[EMAIL PROTECTED] (xxx.xxx.xxx.xxx) 
Jul 13 23:45:11 web rsyncd[26323]: wrote 4335227 bytes  read 295 bytes  
total  size 31683130932 


Is there a way to get both timestamps, stamping localtime?

Thanks

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync.log - two differ timestamps

[Wayne Davison]

On Wed, Jul 13, 2005 at 05:01:55PM -0700, Steve Magee wrote:
 The problem is the timestamp are using localtime and Greenwich time.

This has come up:

http://lists.samba.org/archive/rsync/2005-January/011353.html

... several times before:

https://bugzilla.samba.org/show_bug.cgi?id=2607

Unfortunately, I have not heard of anyone discovering exactly what the
problem is, though it appears to be outside of rsync's control.

One way to work around the problem may be to set use chroot = off in
your rsyncd.conf file.  Does your system use syslogd (the daemon for
syslog() calls)?  If not, try enabling that.  This all works fine on my
Debian Sarge/Etch systems (which use libc6-2.6.2.ds1, just in case
that's pertinent).

..wayne..
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

timestamps

[Juergen Busam]

Hi all!

I'm syncing a windows share from a NetApp filer to a local partition on
my RHEL3 box. I observe the following behavior:

- rsync syncs more than neccessary, files that haven't changed since years
- files that have been synced, aged for an hour or two on the netapp host

Environment:

cifs share mounted to /backup/sync (mount -t smbfs -o
username=user,password=pwd //server/share /backup/sync)

this user has only read rights on the windows share

mirror the mounted cifs share to a local partition (rsync -a
--delete-after /backup/sync/ /dest/dir)

where is my problem? do I use the wrong switches? how is it possible
that the files age a few hours on the source?

Thank you all in advance for your help

Juergen


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: timestamps

[Paul Slootman]

On Wed 25 May 2005, Juergen Busam wrote:
 
 I'm syncing a windows share from a NetApp filer to a local partition on
 my RHEL3 box. I observe the following behavior:
 
 - rsync syncs more than neccessary, files that haven't changed since years
 - files that have been synced, aged for an hour or two on the netapp host

I can't quite parse that last sentence...

 Environment:
 
 cifs share mounted to /backup/sync (mount -t smbfs -o
 username=user,password=pwd //server/share /backup/sync)

Why use CIFS? Netapp understands NFS like no one else... so use NFS!


 this user has only read rights on the windows share
 
 mirror the mounted cifs share to a local partition (rsync -a
 --delete-after /backup/sync/ /dest/dir)

Using rsync over a network drive isn't that useful, rsync is meant to
optimize network traffic between de sender and the receiver, possibly at
the cost of more disk IO. In this case, disk IO on the sender *is*
network traffic...

Use -v (one or more times) to see exactly what's going on.


Paul Slootman
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: timestamps

[Wayne Davison]

On Wed, May 25, 2005 at 09:24:50PM +1000, Juergen Busam wrote:
 how is it possible that the files age a few hours on the source?

If you're running at least 2.6.4, use the -i option to see a summary of
what differences rsync sees between the pairs of files.  For instance, a
summary like this:

f..t foobar.baz

Tells you that the timestamp was different (t) and that the file (f) was
transferred to a local destination ().  Once you know what is different
about the files, you'll be able to investigate how those differences got
there.

..wayne..
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: timestamps

[Juergen Busam]

I actually have to use rsync-2.5.7-5.3E which is included in RHEL3.

I assume that the problem with the aging timestamps results in more
files are transferred...

I've no idea why the source timestamps are changed... Can anyone explain
this?? Or does anyone has an idea why it could happen?

Juergen

Wayne Davison wrote:
 On Wed, May 25, 2005 at 09:24:50PM +1000, Juergen Busam wrote:
 
how is it possible that the files age a few hours on the source?
 
 
 If you're running at least 2.6.4, use the -i option to see a summary of
 what differences rsync sees between the pairs of files.  For instance, a
 summary like this:
 
 
f..t foobar.baz
 
 
 Tells you that the timestamp was different (t) and that the file (f) was
 transferred to a local destination ().  Once you know what is different
 about the files, you'll be able to investigate how those differences got
 there.
 
 ..wayne..
 
 
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: timestamps

[zhijiang tian]

I actually have to use rsync-2.5.7-5.3E which is included in RHEL3.

I assume that the problem with the aging timestamps results in more
files are transferred...

I've no idea why the source timestamps are changed... Can anyone explain
this?? Or does anyone has an idea why it could happen?

Juergen

Wayne Davison wrote:
 On Wed, May 25, 2005 at 09:24:50PM +1000, Juergen Busam wrote:
 
how is it possible that the files age a few hours on the source?
 
 
 If you're running at least 2.6.4, use the -i option to see a summary of
 what differences rsync sees between the pairs of files.  For instance, a
 summary like this:
 
 
f..t foobar.baz
 
 
 Tells you that the timestamp was different (t) and that the file (f) was
 transferred to a local destination ().  Once you know what is different
 about the files, you'll be able to investigate how those differences got
 there.
 
 ..wayne..
 
 
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html



-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Q: rsync - preserve timestamps, but not use it to define files to update

[Eugene Kramer]

 Hello listers,

I do not seem to be able to find a combination of rsync switches that
updates destination directory only if the contents of the files
changes *and* preserves files' original timestamps.

What I've tried so far:

1. --checksum --size-only --ignore-times with or without --times switch.
2. --modify-window=4294967295
3. --size-only w/o --times and with -I (upper case i)

The results: either timestamp gets set whenever there is a difference
in the timestamps between src and dst, or the timestamp of the
transmission gets set for the files in the destination.

rsync 2.6.3, Solaris 2.8

Example of a desired behavior.

I have directories /tmp/a  and /tmp/b with the following files:
/tmp/a:
-rw-r--r--   1 user  group  0 Mar 27 17:22 a
-rw-r--r--   1 user  group  0 Mar 27 17:22 b

/tmp/b:
-rw-r--r--   1 user  group  0 Mar 26 16:44 a
-rw-r--r--   1 user  group  0 Mar 26 16:44 b

I want rsync switches that won't update /tmp/b in above case, but if
/tmp/b is empty, I want /tmp/b/a and /tmp/b/b files to be created with
Mar 27 17:22 timestamp.

Thank you,

Eugene.
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync and timestamps of local files

[Andrew J. Schorr]

On Sat, Mar 08, 2003 at 01:13:17PM -0500, Haisam K. Ido wrote:
 Is there a way to make rsync check the local file system for changes in the files 
 prior to it performing a diff with the remote site?

There is no built-in capability to do this in rsync.   However, you can
implement this yourself.  For example, if you touch a timestamp file
before invoking rsync, then you can use find -newer timestamp to
find files that have changed since the last time you ran rsync.

Then, you can use the --files-from patch to feed the specific list
of files to transfer into rsync.  That patch is available here:

   http://www.clari.net/~wayne/rsync-files-from.patch

Please search the archives for files-from to get more info
on what this patch does.

Cheers,
Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

rsync and timestamps of local files

[Haisam K. Ido]

Is there a way to make rsync check the local file system for changes in the files 
prior to it performing a diff with the remote site?

--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: rsync and timestamps of local files

[bob parker]

On Sun, 9 Mar 2003 05:13, Haisam K. Ido wrote:
 Is there a way to make rsync check the local file system for changes in the
 files prior to it performing a diff with the remote site?

My understanding is that it does nothing if your file(s) has the same 
timestamp and size. Unless you use the -I ( --ignore-times ) option.

hth
Bob

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: rsync and timestamps of local files

[Haisam K. Ido]

thanks Bob.

The timestamp relative to the local first prior to any remote diffing.  Is that what 
you meant?

bob parker wrote:
On Sun, 9 Mar 2003 05:13, Haisam K. Ido wrote:

Is there a way to make rsync check the local file system for changes in the
files prior to it performing a diff with the remote site?


My understanding is that it does nothing if your file(s) has the same 
timestamp and size. Unless you use the -I ( --ignore-times ) option.

hth
Bob
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: rsync and timestamps of local files

[bob parker]

On Sun, 9 Mar 2003 05:45, Haisam K. Ido wrote:
 thanks Bob.

 The timestamp relative to the local first prior to any remote diffing.  Is
 that what you meant?


I think so. It compares the size and timestamp of your local file with the 
remote one of the same name of course. If it all matches nothing is done 
unless you force it.

hth
Bob

 bob parker wrote:
  On Sun, 9 Mar 2003 05:13, Haisam K. Ido wrote:
 Is there a way to make rsync check the local file system for changes in
  the files prior to it performing a diff with the remote site?
 
  My understanding is that it does nothing if your file(s) has the same
  timestamp and size. Unless you use the -I ( --ignore-times ) option.
 
  hth
  Bob
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: rsync and timestamps of local files

[Haisam K. Ido]

bob parker wrote:
On Sun, 9 Mar 2003 05:45, Haisam K. Ido wrote:

thanks Bob.

The timestamp relative to the local first prior to any remote diffing.  Is
that what you meant?


I think so. It compares the size and timestamp of your local file with the 
remote one of the same name of course. If it all matches nothing is done 
unless you force it.
If it does that then it is not doing what I'm asking about.

I need to know if it checks the local files relative to a local log file that it 
keeps of previous rsyncings. If it finds differences between the local file and the 
log file content will it then and only them perform a diff with the remote file with 
the same name?

hth
Bob

bob parker wrote:

On Sun, 9 Mar 2003 05:13, Haisam K. Ido wrote:

Is there a way to make rsync check the local file system for changes in
the files prior to it performing a diff with the remote site?
My understanding is that it does nothing if your file(s) has the same
timestamp and size. Unless you use the -I ( --ignore-times ) option.
hth
Bob
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: rsync and timestamps of local files

[Haisam K. Ido]

Max Bowsher wrote:
Haisam K. Ido wrote:

bob parker wrote:

I think so. It compares the size and timestamp of your local file
with the remote one of the same name of course. If it all matches
nothing is done unless you force it.
If it does that then it is not doing what I'm asking about.

I need to know if it checks the local files relative to a local log
file that it keeps of previous rsyncings. If it finds differences
between the local file and the log file content will it then and only
them perform a diff with the remote file with the same name?


There is no such log file.
Is'nt that inefficient for the network traffic?

Max.
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: rsync and timestamps of local files

[Max Bowsher]

Haisam K. Ido wrote:
 Max Bowsher wrote:
 Haisam K. Ido wrote:
 
 bob parker wrote:
 
 I think so. It compares the size and timestamp of your local file
 with the remote one of the same name of course. If it all matches
 nothing is done unless you force it.
 
 If it does that then it is not doing what I'm asking about.
 
 I need to know if it checks the local files relative to a local log
 file that it keeps of previous rsyncings. If it finds differences
 between the local file and the log file content will it then and
 only them perform a diff with the remote file with the same name?
 
 
 There is no such log file.
 
 Is'nt that inefficient for the network traffic?

Maybe. Depends on the usage scenario.

Max.

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: HELP !!! Problem with file timestamps updating weird during rsync data pull

[Wayne Davison]

On Wed, Oct 16, 2002 at 01:36:10PM -0500, Sean O'Neill wrote:
 The timestamp should match that of the system the data is pulled from right 
 ?  Well, it doesn't from time to time.  The time stamp sometimes gets 
 updated as just Oct 16 2002

This is what most unix systems display for a future date.  I'm guessing
that the clocks on your systems are not in sync -- that the clock on the
receiving end is behind the sending end, which causes files that have
been recently modified on the sender to show up as having future dates
on the receiving system.

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: HELP !!! Problem with file timestamps updating weird during rsync data pull

[Sean O'Neill]

At 12:10 PM 10/16/2002 -0700, Wayne Davison wrote:
On Wed, Oct 16, 2002 at 01:36:10PM -0500, Sean O'Neill wrote:
  The timestamp should match that of the system the data is pulled from 
 right
  ?  Well, it doesn't from time to time.  The time stamp sometimes gets
  updated as just Oct 16 2002

This is what most unix systems display for a future date.  I'm guessing
that the clocks on your systems are not in sync -- that the clock on the
receiving end is behind the sending end, which causes files that have
been recently modified on the sender to show up as having future dates
on the receiving system.

Ahh, excellent point.  These systems aren't as tightly controlled as our 
development systems.  I'm not sure if NTP is setup or setup and running 
correctly.

Thanks, I'll check this out.


--

. ..- -. .. -..- .-. ..- .-.. . ... 
.-- .. -. -... .-.. --- .-- ... -.. .-. --- --- .-.. ...

Sean O'Neill

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Fwd: Re: HELP !!! Problem with file timestamps updating weird during rsync data pull

[Sean O'Neill]

Date: Wed, 16 Oct 2002 14:26:46 -0500
To: Wayne Davison [EMAIL PROTECTED]
From: Sean O'Neill [EMAIL PROTECTED]
Subject: Re: HELP !!! Problem with file timestamps updating weird during 
rsync data pull
Cc: [EMAIL PROTECTED]

At 12:10 PM 10/16/2002 -0700, Wayne Davison wrote:
On Wed, Oct 16, 2002 at 01:36:10PM -0500, Sean O'Neill wrote:
  The timestamp should match that of the system the data is pulled from 
 right
  ?  Well, it doesn't from time to time.  The time stamp sometimes gets
  updated as just Oct 16 2002

This is what most unix systems display for a future date.  I'm guessing
that the clocks on your systems are not in sync -- that the clock on the
receiving end is behind the sending end, which causes files that have
been recently modified on the sender to show up as having future dates
on the receiving system.

Ahh, excellent point.  These systems aren't as tightly controlled as our 
development systems.  I'm not sure if NTP is setup or setup and running 
correctly.

Thanks, I'll check this out.

Yep, NTP was configured to run on my data collection system.  Over 150 
seconds behind the remote systems.  Hopefully Orca will be better behaved now.

Fixed ... Thanks :)


--

. ..- -. .. -..- .-. ..- .-.. . ... 
.-- .. -. -... .-.. --- .-- ... -.. .-. --- --- .-.. ...

Sean O'Neill

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

timestamps, UTC, time zone differences and windows

[Robert Scholten]

Hi all,

I'm having a little trouble using rsync from my windows machine to a Linux
server in a different time zone.  I did some hunting through the
documentation and through the code, but it's not clear to me what rsync
does in this case.

I gather that in principle it uses UTC at both ends.  Will that happen for
a windows client (ssh to a linux box), courtesy of cygwin?

Thanks for any advice.
Rob.


--
Robert Scholten
Eindhoven University of Technology
Physics Department, building N-laag room g2.02
P.O. Box 513, 5600 MB Eindhoven
The Netherlands

Tel:+31 40 247 4242
Mobile: +31 611 430 467
Fax:+31 40 245 6050

email: [EMAIL PROTECTED]
http://www.ph.unimelb.edu.au/~scholten




-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

rsync, smbmount, NT and timestamps

[John N S Gill]

For some time I've been using rsync to sync up some NT file folders and
it has been working like a treat.

I use smbmount to mount the NT shares to linux boxes at each end of the
link and then let rsync do the rest.

Last week the linux boxes were upgraded to redhat 7.1.  I am now using
the following packages:

samba-2.0.7-36
samba-client-2.0.7-36
rsync-2.4.6-2

Since the upgrade i am finding the modify times on the receiving end of
the job are 2 seconds off from the sending end.

When I use 'ls -l --full-time' on the NT shares i see that all the
timestamps on the receiving end have an even number in the seconds
column.  On the sending end I see a mixture of odd and event seconds
(note most of the files on the sending end were created by users running
NT itself).

It looks like smbmount/samba can only set the time on my NT shares to
the nearest 2 seconds, but the problems I'm seeing aren't quite that
simple.

For instance I see thinks like this:

(sending machine)
drwxrwx---1 rems rems  512 Wed Mar 21 09:08:40 2001 X/XYZ

(receiving machine)
drwxrwx---1 rems rems  512 Wed Mar 21 13:08:38 2001 X/XYZ

If it was as simple as having to round to nearest even time then there
should be no problem if the time stamp at the sending end had an even
time.

I'm not sure if this is a bug or a feature + if it is a bug whether it
is rsync, samba or NT that is causing the problem.

The good news is that there is an easy work-around, I have added:

 --modify-window 2

to my rsync options.

However, I'd prefer not to have to do this. 

John

Re: rsync, smbmount, NT and timestamps

[Dave Dykstra]

On Wed, May 02, 2001 at 11:31:31AM +0100, John N S Gill wrote:
 For some time I've been using rsync to sync up some NT file folders and
 it has been working like a treat.
 
 I use smbmount to mount the NT shares to linux boxes at each end of the
 link and then let rsync do the rest.
 
 Last week the linux boxes were upgraded to redhat 7.1.  I am now using
 the following packages:
 
 samba-2.0.7-36
 samba-client-2.0.7-36
 rsync-2.4.6-2
 
 Since the upgrade i am finding the modify times on the receiving end of
 the job are 2 seconds off from the sending end.
 
 When I use 'ls -l --full-time' on the NT shares i see that all the
 timestamps on the receiving end have an even number in the seconds
 column.  On the sending end I see a mixture of odd and event seconds
 (note most of the files on the sending end were created by users running
 NT itself).
 
 It looks like smbmount/samba can only set the time on my NT shares to
 the nearest 2 seconds, but the problems I'm seeing aren't quite that
 simple.
 
 For instance I see thinks like this:
 
 (sending machine)
 drwxrwx---1 rems rems  512 Wed Mar 21 09:08:40 2001 X/XYZ
 
 (receiving machine)
 drwxrwx---1 rems rems  512 Wed Mar 21 13:08:38 2001 X/XYZ
 
 If it was as simple as having to round to nearest even time then there
 should be no problem if the time stamp at the sending end had an even
 time.
 
 I'm not sure if this is a bug or a feature + if it is a bug whether it
 is rsync, samba or NT that is causing the problem.
 
 The good news is that there is an easy work-around, I have added:
 
  --modify-window 2
 
 to my rsync options.
 
 However, I'd prefer not to have to do this. 


If I remember correctly, that's precisely why that option was added.  I'm
not sure why you didn't see the problem before.  Ah yes, see
http://lists.samba.org/pipermail/rsync/2000-July/002503.html
which says it defaults to 2 on Windows.  It would be good if the man page
said it should be 2 when dealing with a FAT filesystem.

- Dave Dykstra

Re: rsync, smbmount, NT and timestamps

[John N S Gill]

 
First thanks very much for the help.


 
 If I remember correctly, that's precisely why that option was added.  I'm
 not sure why you didn't see the problem before.  Ah yes, see
 http://lists.samba.org/pipermail/rsync/2000-July/002503.html
 which says it defaults to 2 on Windows.  

hmmm this doesn't quite explain why I didn't have problems in the past.
I am not running rsync on windows + never have.

The version that i was using prior to the recent upgrade was:

Name: rsync   Relocations: /usr 
Version : 2.4.1   Vendor: Red Hat, Inc.
Release : 2   Build Date: Tue Mar 20:55:12 2000

together with:
rpm -qif /usr/sbin/smbmount
Name: samba   Relocations: /usr 
Version : 2.0.7   Vendor: (none)
Release : 2425Build Date: Wed Apr 26 03:46:34 2000

I think I will take a look at smbmount/samba to see if anything has
changed there.  Its making my head hurt trying to figure out what is
going on.. the timestamps are always off by two seconds and always in
the same direction, so it is as if things are getting rounded down on
read and rounded down again on write.. but it isn't that simple.

Time to read the source i think...

 It would be good if the man page
 said it should be 2 when dealing with a FAT filesystem.

Good news, the man page pretty much says that... at least it says you
need to use --modify-window for FAT:

   --modify-window
  When  comparing  two  timestamps  rsync  treats the
  timestamps as being equal if they  are  within  the
  value  of modify_window. This is normally zero, but
  you may find it useful to  set  this  to  a  larger
  value  in  some  situations.  In  particular,  when
  transferring to/from FAT filesystems  which  cannot
  represent  times  with  a  1 second resolution this
  option is useful.

The file systems I'm dealing with are NT not FAT.. but its the same
vendor :(.

John