[Bug 7249] Add an option to use O_NOATIME

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7249

--- Comment #17 from Wayne Davison  ---
I just tweaked the option name to --open-noatime, which I think is clearer.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7249] Add an option to use O_NOATIME

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7249

Wayne Davison  changed:

   What|Removed |Added

 Resolution|--- |FIXED
 Status|NEW |RESOLVED

--- Comment #16 from Wayne Davison  ---
The --set-noatime option was just added to the main git (along with the
--atimes option for copying access times).

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7249] Add an option to use O_NOATIME

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7249

--- Comment #15 from Michal Ruprich  ---
Thanks Samuel,

now I can enable tests during build in Fedora and keep the noatime option as
well. Still though, it would be good to have upstream approval on this :/

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7249] Add an option to use O_NOATIME

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7249

Samuel Henrique  changed:

   What|Removed |Added

  Attachment #15524|0   |1
is obsolete||

--- Comment #14 from Samuel Henrique  ---
Created attachment 15525
  --> https://bugzilla.samba.org/attachment.cgi?id=15525=edit
Adds --noatime option for 3.1.3

There were two typos in the previous patch, not a big deal
but I updated again to be 100%.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7249] Add an option to use O_NOATIME

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7249

--- Comment #13 from Samuel Henrique  ---
Created attachment 15524
  --> https://bugzilla.samba.org/attachment.cgi?id=15524=edit
Adds --noatime option for 3.1.3

Updated the patch to fix "make test" with the help of
Paul Slootman.

I'm happy to say that we are now using this patch and running the tests
both at build time and in our pipelines (CI) on Debian.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7249] Add an option to use O_NOATIME

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7249

--- Comment #12 from Samuel Henrique  ---
The patch is currently introducing a regression on the target "test" (make
test).

/usr/bin/ld: syscall.o: in function `do_open':
./syscall.c:206: undefined reference to `noatime'

It happens when the target "tls$(EXEEXT)" gets run.

I'm trying to debug it so we can have an updated noatime patch AND run make
test when building rsync. Any help is appreciated.

It looks like currently neither Debian nor Fedora nor Redhat is making use of
make test, I know this is the blocker on the Debian case.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7249] Add an option to use O_NOATIME

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7249

--- Comment #11 from Samuel Henrique  ---
This patch has been being applied to the Debian packaging of rsync since 2014
(5 years now).

As Michal Ruprich said, this patch requires the build to be made in a certain
way as it can lead to undefined references to noatime in syscall.c.

I stumbled upon this when converting the Debian packaging to use debhelper, we
start having this problem when using it.

It would be nice to have this properly upstreamed.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7249] Add an option to use O_NOATIME

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7249

--- Comment #10 from Michal Ruprich  ---
Hi,

I think that ideally you should decide what to do with this. This almost 10
years old bug is still here and it seems no one from the rsync developers paid
any attention to this. If you are not planning to do anything, maybe you should
just close it.

Best regards,
Michal

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7249] Add an option to use O_NOATIME

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7249

--- Comment #9 from Michal Ruprich  ---
Did any of you who uses rsync with noatime patch had problems with running
'make check' command? It seems to me that some of the source files might be
compiling in different order than with a simple make since the build ends up
with undefined reference to `noatime' in the syscall.c file.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7249] Add an option to use O_NOATIME

[just subscribed for rsync-qa from bugzilla via rsync]

https://bugzilla.samba.org/show_bug.cgi?id=7249

--- Comment #8 from Michal Ruprich  ---
Is there any update on this feature? Is there a plan to merge the patch into
master branch of rsync?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7249] Add an option to use O_NOATIME

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=7249

--- Comment #7 from cyril...@gmail.com ---
+1
I'm currently using a locally patched version, because I'm using rsync to
mirror my whole data daily, plus using a deduplicating backup tool weekly that
stores atime (so if rsync sets atime, the metadata are never deduplicated).
It is logical for a backup tool to avoid setting atime, and logical that it is
an option for rsync that may or may not be used as a backup tool.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: O_NOATIME ?

[Theodore Ts'o]

On Wed, Oct 26, 2016 at 11:46:50AM +0200, Paul Slootman wrote:
> Most filesystems on modern linux systems should be mounted with the
> relatime option.

This is the default...

> You can also use noatime as a mount option, but then be sure that no
> application uses the atimes of files; e.g. something like mutt use the
> atime and mtime to determine whether there is a mail file with unread
> mail.

With the latest Linux kernels (4.0+), there is also the "lazytime"
mount option.  This causes the kernel to avoid writing back inodes
that only have "dirty timestamps", until either (a) some other change
is made to the inode, (b) fsync(), syncfs(), or sync() is called, (c)
an undeleted inode is evicted from memory or the file system is
unounted, (d) 24 hours have gone by, or (e) in the case of ext4, if
some other change is made to an inode in the same inode table block as
an inode with a dirty timestamp (so we need to do the disk I/O
anyway).

With lazytime, the timestamp is updated in memory, so stat(2) will
always return the correct timestamp, and in normal practice, the
timestamps will be (eventually) updated on disk.  However if a
timestsamp gets updated multiple times --- for example, a database
file getting updated with O_DIRECT writes, instead of the mtime field
being written out every 30 seconds, we can defer the timestamp updates
and collapse them into a much smaller set of inode table block writes.

Cheers,

- Ted

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: O_NOATIME ?

[Paul Slootman]

On Wed 26 Oct 2016, devz...@web.de wrote:
> 
> since we are using rsync for backing up millions of files in a virtual 
> environment, and most of the virtual machines run on SSD cached storage, i`d 
> be curious how that negatively impacts lifetime of the SSD`s when we do rsync 
> run every night for backup
> 
> my question:
> does rsync normal file comparison run to determine if anything has changed 
> change atime of any files ?
> 
> for me it seems, stat/lstat calls of rsync do NOT modify atime, but i`m not 
> sure under which conditions atime is changed. 

Most filesystems on modern linux systems should be mounted with the
relatime option.

The atime will then only be updated if either the mtime or ctime is
newer than the atime, or if the atime is older than a defined interval.

Note that simply using stat will not update the atime, as the *file*
itself has not been accessed, only its metadata.

There's also the nodiratime option that can be useful, as directories
are read to find what files exist in those directories; and it's
seldomly useful to maintain the atime of a directory.

You can also use noatime as a mount option, but then be sure that no
application uses the atimes of files; e.g. something like mutt use the
atime and mtime to determine whether there is a mail file with unread
mail.


Paul

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

O_NOATIME ?

[devzero]

Hello, 

since we are using rsync for backing up millions of files in a virtual 
environment, and most of the virtual machines run on SSD cached storage, i`d be 
curious how that negatively impacts lifetime of the SSD`s when we do rsync run 
every night for backup

my question:
does rsync normal file comparison run to determine if anything has changed 
change atime of any files ?

for me it seems, stat/lstat calls of rsync do NOT modify atime, but i`m not 
sure under which conditions atime is changed. 

grepping the source for O_NOATIME in rsync3.txt i found :

  - Propagate atimes and do not modify them.  This is very ugly on
Unix.  It might be better to try to add O_NOATIME to kernels, and
call that.

furhermore, apparently there _IS_ O_NOATIME in linux kernels for a while:

http://man7.org/linux/man-pages/man2/open.2.html

O_NOATIME (since Linux 2.6.8)
  Do not update the file last access time (st_atime in the
  inode) when the file is read(2).

  This flag can be employed only if one of the following
  conditions is true:

  *  The effective UID of the process matches the owner UID of
 the file.

  *  The calling process has the CAP_FOWNER capability in its
 user namespace and the owner UID of the file has a mapping
 in the namespace.

  This flag is intended for use by indexing or backup programs,
  where its use can significantly reduce the amount of disk
  activity.  This flag may not be effective on all filesystems.
  One example is NFS, where the server maintains the access
  time.


so, maybe someone likes to comment on NOATIME !? 

maybe it could be useful to make rsync honour O_NOATIME ?

regards
roland

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7249] Add an option to use O_NOATIME

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=7249

--- Comment #6 from don...@gmx.com ---
+1 from here as well.

I understand the arguments about the limited usefulness of the atime field in
general, and I agree with much of it, but to me there are more use cases than
what's mentioned in what I've read so far - such as providing helpful clues for
certain kinds of forensic debugging.  atime has been around for a very long
time (every *NIX variant I've worked on, going back to the 1980's), so even
though its utility may be a bit specialized, there are clearly people who are
surprised to see it ignored by a tool like rsync.  IMO back-up/replication/sync
tools should strive to record or replicate the originals as faithfully as
possible[*], with minimal impact on the contents or meta-data of files being
backed up or replicated.

That is to say, one should ideally be able to use rsync to do things like
moving a directory tree to a new place (file system, machine), or
copying/sync'ing it to a back-up location, and later restoring the originals,
without the user of those files being able to discern any differences in them.

I built rsync with Nicolas George's O_NOATIME patch on the latest Git, and IMO
this change brings rsync one step closer to that ideal (on Linux at least).  It
not only avoids clobbering the atime values for the original files as they're
being read during a first copy operation, but also when they're being
checksummed (on both sides) if one uses the -c option in a later re-sync.

Martin von Gagern: That's the main reason that I think --noatime should be
sent to the remote side.  I agree that in some/many cases, rsync users would
not care about atimes being preserved in the copy.  For backups, supposing
atimes had been copied to the backup machine[*] (perhaps not part of the
scenario you had in mind), they would need to be preserved during re-syncs in
order to be able to available when sync'd back to the original place during a
restore.

One problem with unconditionally sending a new option to the remote side is
that if it isn't supported on the remote side, it fails, whereas I would still
see benefit (for some use cases) in using O_NOATIME on the local side only in
that case.  Not sure how easy it would be to handle that situation.

Aside from that, I would really like to see this enhancement be integrated as
soon as possible.  Since it's an option, and doesn't affect the protocol etc,
it doesn't seem as though it would have any negative impact when it isn't being
used.  Seems like a win-win. :)  Could it be integrated before the next
release?


[*] Which brings us to the --atime patch -- but that's another topic.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7249] Add an option to use O_NOATIME

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=7249

Martin von Gagern martin.vgag...@gmx.net changed:

   What|Removed |Added

   Attachment #9278||review?
  Flags||

--- Comment #5 from Martin von Gagern martin.vgag...@gmx.net 2013-10-13 
09:21:06 UTC ---
Created attachment 9278
  -- https://bugzilla.samba.org/attachment.cgi?id=9278
--noatime option for 3.1.0

I rebased Nicholas' patch to 3.1.0, with modifications as described below.

One thing I'm not sure is whether the --noatime flag should automatically be
forwarded to the server. In a general backup scenario, the client would wish to
preserve atimes, but on the backup server I see no reason to do so. Therefore I
made the forwarding part a comment. Users can still explicitely include the
--noatime in the list of server arguments if they whish this behaviour on both
ends.

I've been using this form on 3.0.9 for quite some time, and would like to see
this included officially one day soon. I hope setting it to “review=?” the way
I did is the way to go for that.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Bug 7249] Add an option to use O_NOATIME

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=7249

Martin von Gagern martin.vgag...@gmx.net changed:

   What|Removed |Added

 CC||martin.vgag...@gmx.net

--- Comment #4 from Martin von Gagern martin.vgag...@gmx.net 2011-12-10 
19:53:32 UTC ---
+1 vote.

Should someone set the review flag to '?' to get Nicolas' patch reviewed? Who?

rsync3.txt in the source tree has this to say about atimes:

Desirable features:
 [...]
 - Propagate atimes and do not modify them.  This is very ugly on
   Unix.  It might be better to try to add O_NOATIME to kernels, and
   call that.

So apparently someone with commit privileges (Martin Pool to be precise)
already has decided that such a feature would be desirable (back in 2001). Does
this add weight to this request and patch?

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

DO NOT REPLY [Bug 7249] Add an option to use O_NOATIME

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=7249


devz...@web.de changed:

   What|Removed |Added

 CC||devz...@web.de




--- Comment #2 from devz...@web.de  2010-08-22 06:15 CST ---
this looks very useful . i vote for it.

seems there is already an atimes.diff in [rsync-patches.git] which has a
similar purpose !?


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug, or are watching the QA contact.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

DO NOT REPLY [Bug 7249] Add an option to use O_NOATIME

[samba-bugs]

https://bugzilla.samba.org/show_bug.cgi?id=7249





--- Comment #3 from nicolas.geo...@normalesup.org  2010-08-22 07:49 CST 
---
They have a similar purpose, but they do not work the same way at all.

My patch (comment #2) politely asks the kernel not to update the atime of
opened files. It only works if the O_NOATIME flags exists on that particular
platform but has no other drawbacks.

AFAIK, the patch on rsync-patches.git explicitly puts back the atime to what it
was. It has the drawback of updating the ctime instead.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug, or are watching the QA contact.
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

O_NOATIME in linux 2.6.8rc2

[mbp]

linux 2.6.8rc2 gained an O_NOATIME open option, which allows you to
make backups without affecting the atime.  Someone might like to add
this into rsync as an option.  It should be pretty trivial.

-- 
Martin


pgpOjwszIPU8F.pgp
Description: PGP signature
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html