[issue47260] os.closerange() can be no-op in a seccomp sandbox
Kyle Evans added the comment: Sure, sounds good to me. The original theory (IIRC, I've slept many times since then :-)) was that we already know first/last are valid and there are no other defined errors, so 'other errors' must be because close_range has started percolating up something from closing individual files. It's been years now and that hasn't happened, even with more recent flag additions. I think it's safe to say it won't, and such a fallback upon error won't put us back into a bogus pre-close_range situation where we're needlessly close()ing a bunch of closed fds. -- ___ Python tracker <https://bugs.python.org/issue47260> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40422] Light refactor: create a common _Py_closerange API
Change by Kyle Evans : -- pull_requests: +21651 pull_request: https://github.com/python/cpython/pull/22680 ___ Python tracker <https://bugs.python.org/issue40422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40422] Light refactor: create a common _Py_closerange API
Change by Kyle Evans : -- pull_requests: +21646 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/22672 ___ Python tracker <https://bugs.python.org/issue40422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40422] Light refactor: create a common _Py_closerange API
Kyle Evans added the comment: Excellent, thank you. -- ___ Python tracker <https://bugs.python.org/issue40422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40422] Light refactor: create a common _Py_closerange API
Kyle Evans added the comment: Would you like that on a distinct issue, or is it ok to reuse this BPO since it's a location improvement of an API just introduced? I've got a local branch now that: 1. moves the suppress IPH stuff into _Py_closerange 2. moves the definition into Python/fileutils.c 3. moves the declaration into Include/fileutils h -- ___ Python tracker <https://bugs.python.org/issue40422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33802] Regression in logging configuration
Change by Kyle Evans : -- nosy: +kevans nosy_count: 8.0 -> 9.0 pull_requests: +21629 pull_request: https://github.com/python/cpython/pull/22651 ___ Python tracker <https://bugs.python.org/issue33802> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in the standard library should be sanitized on fork
Change by Kyle Evans : -- nosy: +kevans nosy_count: 28.0 -> 29.0 pull_requests: +21627 pull_request: https://github.com/python/cpython/pull/22651 ___ Python tracker <https://bugs.python.org/issue6721> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36533] logging regression with threading + fork are mixed in 3.7.1rc2 (deadlock potential)
Change by Kyle Evans : -- nosy: +kevans nosy_count: 8.0 -> 9.0 pull_requests: +21628 pull_request: https://github.com/python/cpython/pull/22651 ___ Python tracker <https://bugs.python.org/issue36533> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40423] Optimization: use close_range(2) if available
Change by Kyle Evans : -- pull_requests: +21624 stage: -> patch review pull_request: https://github.com/python/cpython/pull/22651 ___ Python tracker <https://bugs.python.org/issue40423> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40423] Optimization: use close_range(2) if available
Kyle Evans added the comment: Ah, I will fix this and then submit a PR, thanks... hopefully it returns ENOSYS. -- nosy: +kevans ___ Python tracker <https://bugs.python.org/issue40423> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41013] test_os.test_memfd_create() fails on AMD64 FreeBSD Shared 3.x
Kyle Evans added the comment: Hey koobs, Can you confirm that this is fine now after I implemented SHM_GROW_ON_WRITE? -- ___ Python tracker <https://bugs.python.org/issue41013> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41013] test_os.test_memfd_create() fails on AMD64 FreeBSD Shared 3.x
Kyle Evans added the comment: I can confirm that neither 12 nor 11 will be getting memfd_create; file sealing is a little more complicated to MFC, and I don't want to provide memfd_create in a place where it can't be paired with file sealing in case applications assume they come hand-in-hand and want to use sealing for freezable shm type stuff. -- ___ Python tracker <https://bugs.python.org/issue41013> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41013] test_os.test_memfd_create() fails on AMD64 FreeBSD Shared 3.x
Kyle Evans added the comment: Ah, sorry, I meant to update this- I submitted our fix for review a day or two ago, got approval for commit and will poke koobs to rebuild the FreeBSD -CURRENT buildbot with it after that. -- ___ Python tracker <https://bugs.python.org/issue41013> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41013] test_os.test_memfd_create() fails on AMD64 FreeBSD Shared 3.x
Kyle Evans added the comment: Ah, now that I'm more awake, I see the problem- the write is unsuccessful because I haven't yet implemented grow-on-write. None of the consumers that I had found relied on it, instead choosing to ftruncate() it to the size they need before operating on it. I think the best course of action might be to disable the test on FreeBSD for now (unless you're ok with tossing in a truncate to expand the file), because it might take me a little bit to get around to this one. -- ___ Python tracker <https://bugs.python.org/issue41013> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41013] test_os.test_memfd_create() fails on AMD64 FreeBSD Shared 3.x
Kyle Evans added the comment: (Transcription error beyond the bogus os.exit() :-)) -- ___ Python tracker <https://bugs.python.org/issue41013> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41013] test_os.test_memfd_create() fails on AMD64 FreeBSD Shared 3.x
Kyle Evans added the comment:
I think it's too early to look at this, I'll circle back another time. :-) I
got a minute and extracted the relevant test, but I guess there must have been
some fundamental transcription error:
import os
fd = os.memfd_create("Hi", os.MFD_CLOEXEC)
if fd == -1:
print("boo")
os.exit(1)
if os.get_inheritable(fd):
print("inheritable, boo")
os.exit(1)
with open(fd, "wb", closefd=False) as f:
f.write(b'memfd_create')
if f.tell() != 12:
print("Hork")
os.exit(1)
print("What?")
print("Done")
When I run this with python3.9, I get:
...
3038 100554: shm_open2(SHM_ANON,O_RDWR|O_CLOEXEC,00,0,"memfd:Hi") = 3 (0x3)
3038 100554: fcntl(3,F_GETFD,) = 1 (0x1)
3038 100554: fstat(3,{ mode=-- ,inode=10,size=0,blksize=4096 }) = 0
(0x0)
3038 100554: ioctl(3,TIOCGETA,0x7fffe2a0) ERR#25 'Inappropriate ioctl
for device'
3038 100554: lseek(3,0x0,SEEK_CUR) = 0 (0x0)
3038 100554: lseek(3,0x0,SEEK_CUR) = 0 (0x0)
3038 100554: write(3,"memfd_create",12) = 0 (0x0)
3038 100554: write(3,"memfd_create",12) = 0 (0x0)
3038 100554: write(3,"memfd_create",12) = 0 (0x0)
3038 100554: write(3,"memfd_create",12) = 0 (0x0)
3038 100554: write(3,"memfd_create",12) = 0 (0x0)
(ad infinitum)
So in my local repro, Python is looping forever on successful write() for
reasons I'm not immediately sure of.
--
___
Python tracker
<https://bugs.python.org/issue41013>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41013] test_os.test_memfd_create() fails on AMD64 FreeBSD Shared 3.x
Kyle Evans added the comment: Interesting; I don't quite have time to look at the moment, but what is the test doing that it's ultimately timing out? Our memfd_create is assumed to be compatible, with the exception that we don't yet support HUGETLB (but there are patches in progress for the vm that might make it feasible); in my experience most users of memfd_create weren't really using hugetlb, though. -- ___ Python tracker <https://bugs.python.org/issue41013> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40423] Optimization: use close_range(2) if available
New submission from Kyle Evans : This is dependent on issue40422; the diff on top of that (PR19075) looks like the attached. Effectively, close_range(2) should be preferred at all times if it's available, otherwise we'll use closefrom(2) if available with a fallback to fdwalk(3) or plain old loop over fd range in order of most efficient to least. PR will be sent after issue40422 is resolved. -- components: C API files: cpython-close_range.diff keywords: patch messages: 367531 nosy: kevans91 priority: normal severity: normal status: open title: Optimization: use close_range(2) if available type: enhancement versions: Python 3.9 Added file: https://bugs.python.org/file49097/cpython-close_range.diff ___ Python tracker <https://bugs.python.org/issue40423> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40422] Light refactor: create a common _Py_closerange API
Change by Kyle Evans : -- keywords: +patch nosy: +kevans nosy_count: 1.0 -> 2.0 pull_requests: +19075 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19754 ___ Python tracker <https://bugs.python.org/issue40422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40422] Light refactor: create a common _Py_closerange API
New submission from Kyle Evans : Such an API can be used for both os.closerange and subprocess, re-using much of os_closerange_impl. Pull request enroute. -- components: C API messages: 367530 nosy: kevans91 priority: normal severity: normal status: open title: Light refactor: create a common _Py_closerange API type: enhancement versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue40422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35537] use os.posix_spawn in subprocess
Kyle Evans added the comment: I'll be preparing a patch for our posix_spawn's signal handling. My mistake in my setuid assessment was pointed out to me- it doesn't seem like a highly likely attack vector, but it is indeed possible. If the child errored out, they will indeed be properly reapable at that point due to how vfork is implemented -- any observation to the contrary is to be considered a bug. -- ___ Python tracker <https://bugs.python.org/issue35537> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35537] use os.posix_spawn in subprocess
Kyle Evans added the comment: On Wed, Jan 16, 2019 at 5:47 PM Alexey Izbyshev wrote: > > Hi, > > > As a disclaimer, I'm a FreeBSD developer interested in making sure we're > > doing the right thing here. =) > > > May I ask what the above assessment is based on, and specifically what we > > need to address? > > Hello, Kyle! That assessment is based on my quick and incorrect reading of > posix_spawn source code. I didn't notice that "error" is visible in the > parent due to address space sharing. Sorry for that, and thank you for the > correction. A proper error notification is required for posix_spawn to be > used in subprocess as a replacement for fork/exec, and FreeBSD does it right. Oh, good to hear. =) koobs had pointed this thread out in hopes that we can try to reconcile and figure out what work needs to be done here. =) > While we're here, would you be kind to answer several questions about > posix_spawn on FreeBSD? Most definitely, if I can! > 1) On Linux, without taking additional precautions, signals may be delivered > to a child process created via vfork(). If a signal handler runs in such a > child, it can leave traces of its work in the (shared) memory, potentially > surprising the parent process. To avoid this, glibc[1] and musl[2] disable > all signals (even signals used internally for thread cancellation) before > creating a child, but FreeBSD calls vfork() right away. Is this not > considered a problem, or is something hidden in vfork() implementation? > Right, after glancing over our implementation details- this is indeed a problem here. Our manpage indicates that we only block SIGTTOU for SIGTTIN for children in vfork(), and some light testing seems to indicate that's the case. Signal handlers are inherited from the parent, so that's less than stellar. > 2) Another problem with vfork() is potential sharing of address space between > processes with different privileges (see Rich Felker's post[3] about this and > the previous problem). Is this a valid concern on FreeBSD? My initial read-through of the relevant bits seem to indicate that image activation will cause the child process to be allocated a new process space, so it's looking kind of like a 'no' -- I'm outsourcing the answer to this one to someone more familiar with those bits, though, so I'll get back to you. > 3) Does FreeBSD kernel guarantee that when waitpid(WNOHANG) is called at [4], > the child is ready for reaping? On Linux, it's not always the case[5]. I want to say vfork semantics guarantee it- we got to this point, so the child hit an error and _exit(127). I'm double-checking this one, though, to verify the child's properly marked a zombie before we could possibly reschedule the parent. -- ___ Python tracker <https://bugs.python.org/issue35537> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35537] use os.posix_spawn in subprocess
Kyle Evans added the comment: > * On FreeBSD, if setting posix_spawn() "attributes" or execute posix_spawn() > "file actions" fails, posix_spawn() succeed but the child process exits > immediately with exit code 127 without trying to call execv(). If execv() > fails, posix_spawn() succeed, but the child process exit with exit code 127. Hi, As a disclaimer, I'm a FreeBSD developer interested in making sure we're doing the right thing here. =) May I ask what the above assessment is based on, and specifically what we need to address? As far as I can tell, our implementation is as POSIX describes -- errors processing the file actions or attrs triggers a 127 exit [1][2] which get bubbled up via the return value to posix_spawn [3]. exec failures capture errno at [4] and bubble the error up to the return value of posix_spawn as well via [3]. POSIX explicitly does not require an implementation to use errno for this, only return values, and we seem to have gone the route of not using errno to match OpenSolaris behavior. What do I need to do to reproduce the results for deriving the results seen in the above quote, so that I can fix us and we can also see this improvement? I threw together a minimal C reproducer for posix-spawn on -current (this particular bit being unchanged since FreeBSD 11.x times) and was returned ENOENT for a bad exec and otherwise given a pid for successful exec with a return of 0. [1] https://svnweb.freebsd.org/base/head/lib/libc/gen/posix_spawn.c?view=markup#l214 [2] https://svnweb.freebsd.org/base/head/lib/libc/gen/posix_spawn.c?view=markup#l219 [3] https://svnweb.freebsd.org/base/head/lib/libc/gen/posix_spawn.c?view=markup#l232 [4] https://svnweb.freebsd.org/base/head/lib/libc/gen/posix_spawn.c?view=markup#l225 -- nosy: +kevans ___ Python tracker <https://bugs.python.org/issue35537> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
