[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Martin Panter added the comment: Okay, now at Issue 24675 -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Changes by Martin Panter vadmium...@gmail.com: Removed file: http://bugs.python.org/file39955/stat-times-deprecated.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Martin Panter added the comment: This patch defeats the warnings -- status: closed - open Added file: http://bugs.python.org/file39955/stat-times-deprecated.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Martin Panter added the comment: The commit causes test_os to emit DeprecationWarning warnings, which it didn’t before: [vadmium@localhost cpython]$ hg update 4335d898be59 0 files updated, 0 files merged, 0 files removed, 0 files unresolved [vadmium@localhost cpython]$ ./python -bWdefault -m test test_os [1/1] test_os /media/disk/home/proj/python/cpython/Lib/unittest/case.py:638: DeprecationWarning: stat_float_times() is deprecated function(*args, **kwargs) [× 11 . . .] -- nosy: +vadmium ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
STINNER Victor added the comment: The issue looks to be fixed on Python 3.4, 3.5 and 3.6. I checked quickly buildbots. I close the issue. -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
STINNER Victor added the comment: I applied the fix to Python 3.4 and 3.5 too. Python 2.7 doesn't look to be affected by the issue, I don't see any test copying the timestamp of a file to another, nor test on timestamp using nanosecond resolution. -- versions: -Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Roundup Robot added the comment: New changeset 50ec3fb126dd by Victor Stinner in branch '3.4': Issue #15745: Rewrite os.utime() tests in test_os https://hg.python.org/cpython/rev/50ec3fb126dd New changeset 744c96cd57da by Victor Stinner in branch '3.5': (Merge 3.4) Issue #15745: Rewrite os.utime() tests in test_os https://hg.python.org/cpython/rev/744c96cd57da -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Roundup Robot added the comment: New changeset 4335d898be59 by Victor Stinner in branch 'default': Issue #15745: Rewrite os.utime() tests in test_os https://hg.python.org/cpython/rev/4335d898be59 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
STINNER Victor added the comment: I pushed test_utime.patch with minor updates: * _test_utime_current() now also checks if the FS supports subsecond * use a delta of 20 ms in _test_utime_current(), instead of 10 ms, for Windows I will wait for FreeBSD and Solaris buildbots to see if my changes fixes failures on os.utime() tests. If it's a success, I will backport the change to Python 3.4. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
koobs added the comment: Additionally on koobs-freebsd9, in my home directory (which is on ZFS) The buildbot home directories are on UFS -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
koobs added the comment: Larry: The same two hosts that the FreeBSD Python buildslaves run on :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
R. David Murray added the comment: Note that the shorter patch means that the test is not actually testing what the comments say it is testing, so either the comments should admit we are checking that the result is something close to what we set, or the longer fix should be used so as to continue to use the more rigorous test on platforms that support it. Ideally the latter. -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
koobs added the comment: I have tested both patches (test_os by trent) and almostequaltime by harrison on the default branch, and *both* result in test_os passing. They also resolve the test_utime failure reported in bug 24175 and very likely 16287 (born from this issue) -- versions: +Python 3.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
koobs added the comment: Hmm, that was supposed to be: issue 24175 and very likely issue 16287 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
STINNER Victor added the comment: The resolution of os.utime()+os.stat() depends on two things: - resolution of the timestamp on the filesystem used to run test_os (where TESTFN is written) - resolution of the C internal function used by os.utime() os.utime() can have a resolution of 1 ns (ex: it's usually the case on Linux), whereas the FAT filesystem has as resolution of 2 seconds. os.utime() can have a resolution of 1 us (ex: FreeBSD) whereas the ZFS filesystem has a resolution of 1 ns. Currently, test_os.test_*utime*_ns checks that os.utime() is able to copy the timestamp of a file 1 to a file 2. Problem: we don't know the resolution of the timestamp of the file 2. We can get the resolution of the C internal function used by os.utime(). It is implemented in the attached test_os.patch. But it's much more complex to get the timestamp resolution of the filesystem, in a portable way. Random thoughts: * use a timestamp with a resolution of 1 us, smaller than 2^24 to avoid rounding issues. Example: (atime=1.002003, mtime=4.005006)? * compute the effective utime resolution: call os.utime() with a well known timestamp with a resolution of 1 nanosecond (smaller than 2^24 to avoid rounding issues) and call os.stat() to check which digits were preserved test_os must not depend too much on the filesystem. I don't think that we should implement complex code just to check a simple field in the os.stat_result structure. The first idea (call utime with a fixed number, don't rely on an unknown file timestamp) is probably enough. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
STINNER Victor added the comment: test_utime_ns.patch: rewrite _test_utime_ns(). It now uses constant timestamps for atime and mtime with a resolution of 1 us. The test will fail if the internal function of os.utime() has a resolution of 1 sec (utime() with time_t) of if the resolution of filesystem timestamp is worse than 1 us. In practice on buildbots, it looks like the effective resolution of 1 us (FreeBSD, Solaris), 100 ns (Windows) or 1 ns (Linux). So 1 us should work on all buildbot slaves. test_utime_ns.patch doesn't call os.utime() on directories, only on a regular file. I don't understand the purpose of testing with a directory. Are we testing the OS or Python? -- Added file: http://bugs.python.org/file39688/test_utime_ns.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
STINNER Victor added the comment: almostequaltime.diff is wrong: it allows a different of 10 seconds, whereas the issue is a difference of less than 1000 nanoseconds. test_os.patch looks more correct, but I didn't review the patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
STINNER Victor added the comment: test_utime.patch: a much larger patch which rewrites all unit tests on os.utime(). Changes: * Use a fixed timestamp instead of copying timestamps from an existing file. If the timestamp of the original file can have a resolution of 1 nanosecond, os.utime() rounds to a resolution of 1 us on some platforms (when the C function uses a structure with a resolutionf of 1 us). * Use a fixed timestamp with a resolution of 1 us instead of a resolution of 1 ms. * Remove test_1565150(): it's now redundant with test_utime() and many other test_utime_*() tests * Use self.fname instead of __file__ to check if the filesystem supports subsecond resolution: these two files may be in two different filesystems * test_large_time() now checks the filesystem when it is executed, not when the class is defined. This change is to ensure that we are testing the right filesystem. * replace support.TESTFN with self.dirname for readability * move all os.utime() tests in a new dedicated class * _test_utime_current() now get the system clock using time.time() and tolerate a delta of 10 ms instead of 10 seconds: we may increase the delta because of slow buildbots, but I hope that we can find a value smaller than 10 seconds! * Avoid tricky getattr(), it's no more needed * Merge duplicated test_utime_*() and test_utime_subsecond_*() functions * Test also st_atime on when testing os.utime() on a directory * etc. -- Added file: http://bugs.python.org/file39689/test_utime.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
koobs added the comment: Can a test be made to show a message (similar to a skipIf reason=) mentioning that a reduced precision is being used for certain tests? It would be nice not to have to remember this issue as platform support changes (reads: improves) over time. Not withstanding, it's also apparent that there may be an underlying rounding bug or race condition that ultimately causes some of the assertions in this tests to be false, which is the premise behind Harrisons assertAlmostEqual patch (matching other tests) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Harrison Grundy added the comment: The attached patch just explicitly cuts precision down for the test. It may be worth adding code to set delta to the expected level of precision for a given platform, rather than just universally saying Microseconds are okay. -- nosy: +harrison.grundy Added file: http://bugs.python.org/file39159/almostequaltime.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Changes by koobs koobs.free...@gmail.com: -- components: +Tests keywords: +needs review stage: needs patch - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Changes by Terry J. Reedy tjre...@udel.edu: -- versions: +Python 3.4, Python 3.5 -Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Larry Hastings added the comment: Those interested in this issue might also be interested in #19838, in which I demonstrate that utime(path, NULL) can set mtime to an *earlier* time on ZFS on Linux. -- title: Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS) - Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS) ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Mark Lawrence added the comment: Could we have a formal review of the patch please as Victor seemed fairly happy with it in msg176881. Note that #16287 also refers to this issue. -- nosy: +BreamoreBoy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Changes by Stefan Krah stefan-use...@bytereef.org: -- nosy: -skrah ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Changes by Claudiu.Popa pcmantic...@gmail.com: -- nosy: +Claudiu.Popa ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Changes by Ed Maste carpedd...@gmail.com: -- nosy: +Ed.Maste ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
koobs added the comment: I've moved both of the FreeBSD buildbot slaves off their ZFS-backed home directories and back to good old UFS. I want to ensure FreeBSD support continues to improve, and having slaves get noticed when they fail or regress with ongoing development is a big part of that. This is hard to achieve without movement on this issue, either in the form of a conditional skip, workaround or ultimate resolution -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
koobs added the comment: I'm happy to move them back upon request, or create a FreeBSD/ZFS buildslave specially for the job, just let me know. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
koobs added the comment: There's some work that's been in the FreeBSD bleachers since Jul 2012 to add futimens() and utimensat(), with some recent activity: RFC: futimens(2) and utimensat(2) - Jul 2012 http://lists.freebsd.org/pipermail/freebsd-arch/2012-February/012409.html RFC: futimens(2) and utimensat(2) - Jan 2013 http://lists.freebsd.org/pipermail/freebsd-arch/2013-January/013903.html I've also recently been made aware of a vfs.timestamp_precision sysctl and tested adjusting it from it's default of 0 - 3, without success: sysctl -d vfs.timestamp_precision vfs.timestamp_precision: File timestamp precision (0: seconds, 1: sec + ns accurate to 1/HZ, 2: sec + ns truncated to ms, 3+: sec + ns (max. precision)) I'll do my best at this end to encourage the above implementation to be committed and request merges to other branches of FreeBSD (likely just 9-STABLE) In the meantime however, what can be done in the short-term to either tweak the tests so they pass or account for the difference in implementations? The current test failures on the buildbots make it difficult at best to ensure core developers are being notified or exposed to other regressions and issues that have cropped up in recent months. -- nosy: +koobs ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Changes by Jesús Cea Avión j...@jcea.es: -- nosy: +jcea ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
STINNER Victor added the comment: I didn't read the whole issue, but the following code makes sense. +try: +import posix +except ImportError: +# Windows has nanosecond utime resolution. +UTIME_EPSILON = 2e-9 +else: +import sysconfig +if 'HAVE_UTIMENSAT' in posix._have_functions: +UTIME_EPSILON = 2e-9 +elif 'HAVE_UTIMES' in sysconfig.get_config_vars(): +UTIME_EPSILON = 2e-6 +else: +UTIME_EPSILON = 1.0 Windows doesn't really have nanosecond resolution: the common Windows unit is 100 ns. See GetFileInformationByHandle() documentation: its BY_HANDLE_FILE_INFORMATION structure has FILETIME fields, and FILETIME uses the 100 ns unit. We might expose this unit (UTIME_EPSILON, in the os module maybe?) as we done for time function with time.get_clock_info(). -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Trent Nelson added the comment: On Tue, Oct 16, 2012 at 10:46:34PM -0700, Trent Nelson wrote: Trent Nelson added the comment: Thanks for the feedback Larry; yeah that patch definitely wasn't intended to be production quality -- more of a proof of concept. I agree with your points, they'll be factored into the next patch. However, I'm absolutely baffled by the Solaris 10 failure. The more I looked into it, the weirder it got. The issue is always the same: self.assertEqual(attr(st0, st_mtime), attr(st1, st_mtime)) AssertionError: 1347752941.275297 != 1347752941.275296 That is, test_utime() always results in a st1.st_mtime that is off-by-1 from st0.st_mtime. The precision is well within the nanasecond resolution offered by utimensat, so it doesn't appear to be the same issue experienced by other platforms. I've concluded that the problem on Solaris is actually unrelated to the original failures on FreeBSD and NetBSD that this issue was raised for (where os.stat() returns nanosecond resolution but os.utime() only accepts microsecond). I've raised a separate bug for this issue: #16287. -- title: Numerous utime ns tests fail on FreeBSD w/ ZFS - Numerous utime ns tests fail on FreeBSD w/ ZFS (update:and NetBSD w/ FFS, Solaris w/ UFS) ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Larry Hastings added the comment: I don't know POSIX / UNIX all that well. Does it require that a stat call updates atime? Because that's one of those how does it ever work head-scratchers. (Maybe everybody always disables atime these days? I don't think I do.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Antoine Pitrou added the comment: I don't know POSIX / UNIX all that well. Does it require that a stat call updates atime? Because that's one of those how does it ever work head-scratchers. (Maybe everybody always disables atime these days? I don't think I do.) I think most Linux distributions disable it for you; or, rather, they use the relatime mount option which is a smart cheat: relatime Update inode access times relative to modify or change time. Access time is only updated if the previous access time was earlier than the current modify or change time. (Similar to noatime, but doesn't break mutt or other applications that need to know if a file has been read since the last time it was modified.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Trent Nelson added the comment: Re: how did it ever work... on Solaris, because of the st_mtime failure, it doesn't even get a chance to fail on the subsequent st_atime. I suspect the only platform that's exercised the utimensat() to date is Linux, and either a) os.stat() doesn't affect atime on Linux, b) everyone has atime disabled, c) the two stat calls happen quick enough that no measurable difference is observed against st_atime. As for POSIX: http://pubs.opengroup.org/onlinepubs/9699919799/toc.htm The fstat() function shall update any time-related fields (as described in XBD File Times Update ), before writing into the stat structure. The referenced section: 4.8 File Times Update Each file has three distinct associated timestamps: the time of last data access, the time of last data modification, and the time the file status last changed. These values are returned in the file characteristics structure struct stat, as described in sys/stat.h . Each function or utility in POSIX.1-2008 that reads or writes data (even if the data does not change) or performs an operation to change file status (even if the file status does not change) indicates which of the appropriate timestamps shall be marked for update. If an implementation of such a function or utility marks for update one of these timestamps in a place or time not specified by POSIX.1-2008, this shall be documented, except that any changes caused by pathname resolution need not be documented. For the other functions or utilities in POSIX.1-2008 (those that are not explicitly required to read or write file data or change file status, but that in some implementations happen to do so), the effect is unspecified. An implementation may update timestamps that are marked for update immediately, or it may update such timestamps periodically. At the point in time when an update occurs, any marked timestamps shall be set to the current time and the update marks shall be cleared. All timestamps that are marked for update shall be updated when the file ceases to be open by any process or before a fstat(), fstatat(), fsync(), futimens(), lstat(), stat(), utime(), utimensat(), or utimes() is successfully performed on the file. Other times at which updates are done are unspecified. Marks for update, and updates themselves, shall not be done for files on read-only file systems; see Read-Only File System . The resolution of timestamps of files in a file system is implementation-defined, but shall be no coarser than one-second resolution. The three timestamps shall always have values that are supported by the file system. Whenever any of a file's timestamps are to be set to a value V according to the rules of the preceding paragraphs of this section, the implementation shall immediately set the timestamp to the greatest value supported by the file system that is not greater than V. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Trent Nelson added the comment: Here's a thought... why not alter the test to work with fixed times and separate the atime tests from the mtime tests. For atime, we can set utime(filename, (0.0, ...)) and see if a subsequent os.stat() returns st_atime as 0.0 -- that'll tell us whether or not atime is affected. For the mtime tests, rather than having a variable UTIME_EPSILON, just have fixed dates that match the expected precision of the underlying platform? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Trent Nelson added the comment: I've figured out what the primary problem is on these platforms: os.stat() returns st_mtime and st_atime values with nanosecond resolution, but without a corresponding utimensat(), we can only affect time with microsecond precision via utimes(). Therefore, the following logic is faulty: def _test_utime(self, filename, attr, utime, delta): # Issue #13327 removed the requirement to pass None as the # second argument. Check that the previous methods of passing # a time tuple or None work in addition to no argument. st0 = os.stat(filename) # Doesn't set anything new, but sets the time tuple way utime(filename, (attr(st0, st_atime), attr(st0, st_mtime))) # Setting the time to the time you just read, then reading again, # should always return exactly the same times. st1 = os.stat(filename) self.assertEqual(attr(st0, st_mtime), attr(st1, st_mtime)) self.assertEqual(attr(st0, st_atime), attr(st1, st_atime)) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Larry Hastings added the comment: Is there a different utime family function on these platforms that *can* write atime/mtime with ns resolution? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Trent Nelson added the comment: There doesn't appear to be on FreeBSD. Although, on Solaris, -D__EXTENSIONS__ opens up access to utimensat() (at least on 11), so I'll factor that into configure.ac. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Trent Nelson added the comment: This patch (surprisingly) seems to do the job quite nicely: diff -r 1280b38fe583 Lib/test/test_os.py --- a/Lib/test/test_os.py Tue Oct 16 23:14:03 2012 +1000 +++ b/Lib/test/test_os.py Tue Oct 16 21:25:36 2012 + @@ -40,6 +40,20 @@ or (st.st_mtime != st[8]) or (st.st_ctime != st[9])) +try: +import posix +except ImportError: +# Windows has nanosecond utime resolution. +UTIME_EPSILON = 2e-9 +else: +import sysconfig +if 'HAVE_UTIMENSAT' in posix._have_functions: +UTIME_EPSILON = 2e-9 +elif 'HAVE_UTIMES' in sysconfig.get_config_vars(): +UTIME_EPSILON = 2e-6 +else: +UTIME_EPSILON = 1.0 + # Detect whether we're on a Linux system that uses the (now outdated # and unmaintained) linuxthreads threading library. There's an issue # when combining linuxthreads with a failed execv call: see @@ -312,18 +326,32 @@ st0 = os.stat(filename) # Doesn't set anything new, but sets the time tuple way utime(filename, (attr(st0, st_atime), attr(st0, st_mtime))) -# Setting the time to the time you just read, then reading again, -# should always return exactly the same times. st1 = os.stat(filename) -self.assertEqual(attr(st0, st_mtime), attr(st1, st_mtime)) -self.assertEqual(attr(st0, st_atime), attr(st1, st_atime)) + +def _t(left, right, attr, name): +l = attr(left, name) +r = attr(right, name) +if isinstance(l, int): +assert isinstance(r, int) +l = l / 1e9 +r = r / 1e9 +return abs(l - r) + +epsilon = UTIME_EPSILON +self.assertLess(_t(st0, st1, attr, st_mtime), epsilon) +self.assertLess(_t(st0, st1, attr, st_atime), epsilon) + # Set to the current time in the old explicit way. os.utime(filename, None) st2 = os.stat(support.TESTFN) # Set to the current time in the new way os.utime(filename) st3 = os.stat(filename) -self.assertAlmostEqual(attr(st2, st_mtime), attr(st3, st_mtime), delta=delta) +self.assertAlmostEqual( +attr(st2, st_mtime), +attr(st3, st_mtime), +delta=delta +) def test_utime(self): def utime(file, times): test_os passes on FreeBSD, Linux and Mac OS X with that applied. However, the Solaris 10/SPARC box still fails: AssertionError: 9.5367431640625e-07 not less than 2e-09 But it appears that build is actually picking up utimensat(), which makes the Solaris failures unrelated to the FreeBSD/NetBSD ones. I'll do some more investigating. Might warrant a separate issue. Larry: thoughts on the test_os.patch as is? -- keywords: +patch Added file: http://bugs.python.org/file27599/test_os.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Larry Hastings added the comment: You're a sneaky, naughty bunny. posix._have_functions indeed! I guess your back was to the wall. 1) If I follow your code correctly, when one has utimensat, the assertLess calls using _t consider st0.st_mtime_ns st1.st_mtime_ns even if the left operand is (2e18-1) greater than the right. It divides both operands by 1e9, then checks whether their delta is 2e9. In nanoseconds, 2e18-1 is roughly 63.5 years. The case where one doesn't have utimensat but do have utimens is similarly generous. Little wonder the test now passes! Could you narrow down the range a little? 2) I'd prefer if you amended the comment rather than remove it, perhaps adding # (assuming your read and write functions both take the same units, # which isn't true on all platforms--some Solaris and FreeBSD versions # can read ns time but can only write ms time) 3) Could you please find better names than _t, l, and r? 4) In the future, please upload a patch as an attached file rather than as an inline comment. This will permit me to use the bug tracker's built-in Reitveld to examine the diff, which is more comfortable and permits inline comments. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Trent Nelson added the comment: Thanks for the feedback Larry; yeah that patch definitely wasn't intended to be production quality -- more of a proof of concept. I agree with your points, they'll be factored into the next patch. However, I'm absolutely baffled by the Solaris 10 failure. The more I looked into it, the weirder it got. The issue is always the same: self.assertEqual(attr(st0, st_mtime), attr(st1, st_mtime)) AssertionError: 1347752941.275297 != 1347752941.275296 That is, test_utime() always results in a st1.st_mtime that is off-by-1 from st0.st_mtime. The precision is well within the nanasecond resolution offered by utimensat, so it doesn't appear to be the same issue experienced by other platforms. I'll have to break into the debugger again and see what's going on. Side note: I noticed this comment/code just above _test_utime(): def test_utime_dir(self): delta = 100 st = os.stat(support.TESTFN) # round to int, because some systems may support sub-second # time stamps in stat, but not in utime. os.utime(support.TESTFN, (st.st_atime, int(st.st_mtime-delta))) st2 = os.stat(support.TESTFN) self.assertEqual(st2.st_mtime, int(st.st_mtime-delta)) That... seems to (albeit vaguely) describe what's going on here with Solaris 10. I also noticed support.TESTFN is actually a directory in this test case. Again, I'm not sure how that fits in with test_utime_dir() versus a second invocation of _test_utime(support.TESTFN) from test_utime(). test_utime_dir() is ultra-lax, test_utime() is ultra-strict, I'm not sure which one should be updated to match the other. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Trent Nelson added the comment: Oh, and another quirk I noticed yesterday. I usually religiously disable atime on all my filesystems. For whatever reason, it's not disabled on this Solaris 10 box. Turns out os.stat() was updating st_atime, which kind of throws a spanner in the works for all our st_atime tests in _test_utime() -- we call os.stat() after utime() to check that our atime update worked -- but the stat call results in another st_atime update. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15745] Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS)
Changes by Trent Nelson tr...@snakebite.org: -- title: Numerous utime ns tests fail on FreeBSD w/ ZFS - Numerous utime ns tests fail on FreeBSD w/ ZFS (update: and NetBSD w/ FFS, Solaris w/ UFS) ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15745 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
