[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x: Linux VFS/XFS bug

STINNER Victor added the comment: It seems like AIX also has a kernel issue with timestamp after year 2038. The year 2107 is stored as year 1972. test_add_file_after_2107 (test.test_zipfile.StoredTestsWithSourceFile) ... skipped 'Linux VFS/XFS kernel bug detected:

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x: Linux VFS/XFS bug

miss-islington added the comment: New changeset 2b675f0c8fd96f61977f6dc636f44fbd5587b6b3 by Miss Islington (bot) in branch '3.8': bpo-39460: Fix test_zipfile.test_add_file_after_2107() (GH-18247) https://github.com/python/cpython/commit/2b675f0c8fd96f61977f6dc636f44fbd5587b6b3 --

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x: Linux VFS/XFS bug

Change by miss-islington : -- pull_requests: +17630 pull_request: https://github.com/python/cpython/pull/18253 ___ Python tracker ___

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x: Linux VFS/XFS bug

STINNER Victor added the comment: New changeset 3cb49b62e61208efcefbc04414e769fc173f205d by Victor Stinner in branch 'master': bpo-39460: Fix test_zipfile.test_add_file_after_2107() (GH-18247) https://github.com/python/cpython/commit/3cb49b62e61208efcefbc04414e769fc173f205d --

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x: Linux VFS/XFS bug

STINNER Victor added the comment: The kernel bug is now tracked at https://bugzilla.redhat.com/show_bug.cgi?id=1795576 I wrote PR 18247 to skip the test. I suggest to leave this issue is open until the kernel is fixed: until the test is no longer skipped. -- title: test_zipfile:

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

Change by STINNER Victor : -- keywords: +patch pull_requests: +17626 stage: -> patch review pull_request: https://github.com/python/cpython/pull/18247 ___ Python tracker ___

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: I managed to reproduce the issue on Fedora 31 on a XFS filesystem: $ touch testfn $ strace -o trace python3 -c 'import os; os.utime("testfn", (4386268800, 4386268800))' $ grep ^utimensat trace utimensat(AT_FDCWD, "testfn", [{tv_sec=4386268800, tv_nsec=0} /*

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

Florian Weimer added the comment: I believe you might be observing an XFS limitation in combination with a Linux VFS bug. On disk, XFS only supports 32-bit timestamps: typedef struct xfs_timestamp { __be32 t_sec; /* timestamp seconds */ __be32

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: I failed to reproduce the bug on 3 machines. My x86-64 Fedora VM: * Linux 5.5.0-0.rc6.git3.1.fc32.x86_64 * glibc-2.30.9000-31.fc32.x86_64 * gcc-10.0.1-0.5.fc32.x86_64 * utimensat() available * I tested ext4 (/home) and tmpfs (/tmp) ppc64le mock: * Linux

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: Next question: does the issue come from the glibc or the filesystem? Attached mtime.c should ease debug. If you reproduce the issue, please test different filesystem and report which filesystems you tested. To detect the filesystem used by a directory, I'm

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: Summary: * test_zipfile.test_add_file_after_2107() fails on Fedora Rawhide 3.x * Bug seen on Python 3.8 and Python 3.9 * Bug seen on s390x and x86-64 architectures * It seems like the bug occurs in os.utime() with atime=mtime=4386268800 => stat returns

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: > I failed to reproduce the issue on s390x Fedora Rawhide, Linux kernel > 5.5.0-0.rc4.git2.1.fc32.s390x, glibc-2.30.9000-31.fc32.s390x. I still cannot reproduce after a "dnf upgrade -y": * Linux 5.5.0-0.rc6.git3.1.fc32.s390x * glibc-2.30.9000-31.fc32.s390x

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: I failed to reproduce the issue on s390x Fedora Rawhide, Linux kernel 5.5.0-0.rc4.git2.1.fc32.s390x, glibc-2.30.9000-31.fc32.s390x. I tested filesystem nfs and tmpfs. test_add_file_after_2107() pass. [vstinner@devel10 ~]$ uname -r

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: glibc commit adding support for utimensat_time64() syscall: https://sourceware.org/git/?p=glibc.git;a=commit;h=f5b6fd258b6dd520403a20024e58cb491aca4cbd See also: https://sourceware.org/glibc/wiki/Y2038ProofnessDesign --

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: I'm now curious which syscalls are used by the glibc to implement utimensat() and stat(). On a ppc64le Fedora Rawhide running Linux kernel 5.4.8-200.fc31.ppc64le with glibc 2.30.9000, I get: $ sudo dnf install -y strace # if neeeded $ strace -o trace

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: > [dje@rawhide ~]$ python3 -c 'import os; os.utime("testfn", (4386268800, > 4386268800))' > [dje@rawhide ~]$ stat testfn > (...) > Modify: 2038-01-18 22:14:07.0 -0500 Oh. It means that os.utime() replaces mtime=4386268800 with a value close to

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

David Edelsohn added the comment: In utime_stat_localtime.py, "os.stat (sec)" is the result of os.utime. In utime_stat_localtime2.py "os.stat (sec int)" is the result of os.stat. -- ___ Python tracker

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

David Edelsohn added the comment: [dje@rawhide ~]$ touch testfn [dje@rawhide ~]$ python3 -c 'import os; os.utime("testfn", (4386268800, 4386268800))' [dje@rawhide ~]$ stat testfn File: testfn Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: fd00h/64768d

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: """ $ ./python utime_stat_localtime2.py os.utime (sec): 4386268800 os.stat (sec int): 2147483647 os.stat (sec float): 2147483647.0 os.stat (ns): 21474836470 """ It doesn't make sense !? In msg360916, you had: os.stat (sec): 4386268800 <=

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

David Edelsohn added the comment: Not -O3, but it's calling PyLong_FromLongLong on s390x as well 0x011ca524 <+28>:brasl %r14,0x10649b0 -- ___ Python tracker

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

David Edelsohn added the comment: $ ./python utime_stat_localtime2.py os.utime (sec): 4386268800 os.stat (sec int): 2147483647 os.stat (sec float): 2147483647.0 os.stat (ns): 21474836470 -- ___ Python tracker

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

Change by STINNER Victor : -- title: test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x: time.localtime() limited to year 2038 -> test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x ___ Python

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x: time.localtime() limited to year 2038

STINNER Victor added the comment: David Edelsohn: > Output on s390x Fedora Rawhide: > (...) > os.stat (sec): 4386268800 > os.stat (ns): 21474836470 Oh wait, ns != sec * 10**9 here. "2147483647" is 2**31-1 (INT_MAX). It looks like a bug in fill_time() of Modules/posixmodule.c. I

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x: time.localtime() limited to year 2038

David Edelsohn added the comment: $ ./python Python 3.9.0a3+ (heads/master:aabdeb766b, Jan 28 2020, 13:50:48) [GCC 10.0.1 20200121 (Red Hat 10.0.1-0.4)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import time >>> time.localtime(2**31)

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x: time.localtime() limited to year 2038

STINNER Victor added the comment: Notes on the glibc: * localtime() calls __tz_convert() * localtime() and __tz_convert() have been modified recently to support 64-bit timestamp * __tz_convert() is implemented in time/tzset.c * localtime() is implemented in time/localtime.c --

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: > s390x Fedora Rawhide 3.x: > https://buildbot.python.org/all/#/builders/323/builds/6 I also checked the configure: time_t size is 64-bit on this system, so it should not be impacted on the glibc/Linux kernel work on the year 2038 bug. --

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: I don't see anything about localtime() or the year 2038 bug in glibc 2.30 release notes: https://lwn.net/Articles/795127/ I'm aware of work in the Linux kernel and glibc for the year 2038 bug, but for 32-bit systems: https://lwn.net/Articles/776435/

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: > os.stat (sec): 4386268800 > ... > localtime: (2038, 1, 18, 22, 14, 7) Aha, localtime() behavior changed for timestamp larger than 2**31. What is the version of the glibc package? > s390x Fedora Rawhide 3.x: >

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

David Edelsohn added the comment: Output on s390x Fedora Rawhide: $ ./python utime_stat_localtime.py os.utime (sec): 4386268800 os.stat (sec): 4386268800 os.stat (ns): 21474836470 stat==utime? False localtime: (2038, 1, 18, 22, 14, 7) --

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

STINNER Victor added the comment: I suspect that the timestamp set by os.utime() is not the one seen by os.stat(). Or maybe time.localtime() behavior changed. Try attached utime_stat_localtime.py script. Output on Fedora 31 in my home directory which uses btrfs: $ python3.7

[issue39460] test_zipfile: test_add_file_after_2107() sometimes fails on Fedora Rawhide 3.x

Miro Hrončok added the comment: I think this happens "randomly" regardless of the architecture. I've seen it on x86_64 as well. -- title: test_zipfile: test_add_file_after_2107() fails on s390x Fedora Rawhide 3.x -> test_zipfile: test_add_file_after_2107() sometimes fails on Fedora