I'm guessing that the usage of getdents was intentional: I've used this
system call in the same situation (thousands of entries in a directory,
inside a NFS exported directory) to avoid calling stat() in each entry
returned by getdents.
It is indead a "low level" system call, and usually readdir is the
preferred method, but might be too slow for your usage case (in my case
it was, it took hours to read the directory while with getdents took
minutes).
It doesn't seems a problem with OpenBSD, but with Python code. Looks
like some additional logic will be required to address the differences
betweens the UNIX-like OSs and their implementation of getdents.
On the other hand, by the error message the Python code is passing a
wrong argument, maybe it is a bug instead of a portability issue?
Em 25/07/2020 20:50, Aaron Miller escreveu:
> Hi all,
>
> I am getting a stacktrace from the borg command in the borgbackup
> package while checking a backup (see bottom of email for full
> output, since it's verbose). The relevant part is this:
>
> filenames = os.listdir(os.path.join(data_path, dir))
> OSError: [Errno 22] Invalid argument:
> '/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'
>
> This is same error is reproducible with a test Python 3.8 program:
>
> #!/usr/bin/env python
>
> import os
> os.listdir('/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/')
>
> Running ktrace & kdump reveals the error is from calling
> getdents(2):
>
> 76903 python3.8
> CALL open(0x1ec7f06de3b0,0x30000<O_RDONLY|O_CLOEXEC|O_DIRECTORY>)
> 76903 python3.8
> NAMI "/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/"
> 76903 python3.8 RET open 3
> [...]
> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
> 76903 python3.8 RET getdents 16384/0x4000
> [...]
> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
> 76903 python3.8 RET getdents 16384/0x4000
> [...]
> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
> 76903 python3.8 RET getdents 16384/0x4000
> [...]
> 76903 python3.8 CALL getdents(3,0x1ec7c9257000,0x4000)
> 76903 python3.8 RET getdents -1 errno 22 Invalid argument
>
> Looking at the man page for getdents(2), I found it interesting
> that it says this call "is not a portable interface and should not
> be used directly by applications" and it recommends using
> readdir(3) instead.
>
> To give you a rough idea of the number of files and filename sizes
> in this directory:
>
> $ ls /mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12/ | wc
> 1534 1534 10738
>
> Where does the problem lie -- the upstream Python code, the
> OpenBSD-specific patches in its port definition, or somewhere
> else? And in case it matters, this is a -current amd64 system,
> with "sysupgrade -s" executed on 7/15.
>
> Thank you,
> Aaron Miller
>
> --
> Exception ignored in: <function Repository.__del__ at
> 0x1e17e13fd310>
> Traceback (most recent call last):
> File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 180, in __del__
> assert False, "cleanup happened in Repository.__del__"
> AssertionError: cleanup happened in Repository.__del__
> Local Exception
> Traceback (most recent call last):
> File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
> line 4565, in main
> exit_code = archiver.run(args)
> File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
> line 4497, in run
> return set_ec(func(args))
> File "/usr/local/lib/python3.8/site-packages/borg/archiver.py",
> line 161, in wrapper
> with repository:
> File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 190, in __enter__
> self.open(self.path, bool(self.exclusive),
> lock_wait=self.lock_wait, lock=self.do_lock)
> File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 450, in open
> segment = self.io.get_latest_segment()
> File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 1253, in get_latest_segment
> for segment, filename in self.segment_iterator(reverse=True):
> File "/usr/local/lib/python3.8/site-
> packages/borg/repository.py", line 1241, in segment_iterator
> filenames = os.listdir(os.path.join(data_path, dir))
> OSError: [Errno 22] Invalid argument:
> '/mnt/thinkpad_void_obsd_borg/thinkpad.borg/data/12'
>
> Platform: OpenBSD millipede.iforgotmy.name 6.7 GENERIC.MP#348
> amd64
> Borg: 1.1.13 Python: CPython 3.8.3 msgpack: 0.5.6
> PID: 31745 CWD: /mnt/thinkpad_void_obsd_borg
> sys.argv: ['/usr/local/bin/borg', 'check', 'thinkpad.borg']
> SSH_ORIGINAL_COMMAND: None
>
