[issue11382] some posix module functions unnecessarily release the GIL
Roundup Robot devnull@devnull added the comment: New changeset eb7297fd5840 by Antoine Pitrou in branch 'default': Issue #11382: Trivial system calls, such as dup() or pipe(), needn't http://hg.python.org/cpython/rev/eb7297fd5840 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
Antoine Pitrou pit...@free.fr added the comment: Sorry for the delay, I had forgotten about this issue. Thank you for the patch! -- resolution: - fixed stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
Charles-Francois Natali neolo...@free.fr added the comment: Is there anything I can do to help this move forward ? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
Changes by Terry J. Reedy tjre...@udel.edu: -- type: - performance versions: +Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
New submission from Charles-Francois Natali neolo...@free.fr: Some posix module functions unnecessarily release the GIL. For example, posix_dup, posix_dup2 and posix_pipe all release the GIL, but those are non-blocking syscalls (the don't imply any I/O, only modifying the process file descriptors table). This leads to the famous convoy effect (see http://bugs.python.org/issue7946). For example: $ cat /tmp/test_dup2.py import os import threading import sys import time def do_loop(): while True: pass t = threading.Thread(target=do_loop) t.setDaemon(True) t.start() f = os.open(sys.argv[1], os.O_RDONLY) for i in range(4, 1000): os.dup2(f, i) Whith GIL release/acquire: $ time ./python /tmp/test_dup2.py /etc/fstab real0m5.238s user0m5.223s sys 0m0.009s $ time ./python /tmp/test_pipe.py real0m3.083s user0m3.074s sys 0m0.007s Without GIL release/acquire: $ time ./python /tmp/test_dup2.py /etc/fstab real0m0.094s user0m0.077s sys 0m0.010s $ time ./python /tmp/test_pipe.py real0m0.088s user0m0.074s sys 0m0.008s -- title: some posix module functions - some posix module functions unnecessarily release the GIL ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
Changes by Eric Smith e...@trueblade.com: -- components: +Interpreter Core nosy: +eric.smith, pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: On Windows at least, these functions call malloc() and DuplicateHandle(). -- nosy: +amaury.forgeotdarc ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
Charles-Francois Natali neolo...@free.fr added the comment:
I didn't even know that Windows had such calls.
But anyway, if we start releasing the GIL around each malloc call, then it's
going to get really complicated:
static PyObject *
posix_geteuid(PyObject *self, PyObject *noargs)
{
return PyLong_FromLong((long)geteuid());
}
PyLong_FromLong - _PyLong_New - PyObject_MALLOC which can call malloc.
As for DuplicateHandle, I assume it's as fast as Unix's dup(2).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11382
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
Amaury Forgeot d'Arc amaur...@gmail.com added the comment: What are you trying to achieve? Do you have loops which contain no other syscall than os.dup2()? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
Charles-Francois Natali neolo...@free.fr added the comment: Well, those are contrived examples showing the effect of the convoy effect induced by those unneeded GIL release/acquire: releasing and re-acquiring the GIL comes with a cost (e.g. under Linux, futex are really fast in the uncontended case since handled in use space but much slower when there's contention), and subverts the OS scheduling policy (forcing the thread to drop/re-acquire the GIL make the thread block after having consumed a small amount of its time slice and increases the context switching rate). I think that releasing and re-acquiring the GIL should only be done around potentially blocking calls. Do you have loops which contain no other syscall than os.dup2()? No, but it's not a reason for penalizing threads that use dup, dup2 or pipe. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
Antoine Pitrou pit...@free.fr added the comment: Well, those are contrived examples showing the effect of the convoy effect induced by those unneeded GIL release/acquire: releasing and re-acquiring the GIL comes with a cost (e.g. under Linux, futex are really fast in the uncontended case since handled in use space but much slower when there's contention), and subverts the OS scheduling policy (forcing the thread to drop/re-acquire the GIL make the thread block after having consumed a small amount of its time slice and increases the context switching rate). I think that releasing and re-acquiring the GIL should only be done around potentially blocking calls. Do you want to propose a patch? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
Charles-Francois Natali neolo...@free.fr added the comment: Do you want to propose a patch? Sure, if removing those calls to Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS seems reasonable (I might haved missed something obvious). Just to be clear, I'm not at all criticizing the current GIL implementation, there's been a great work done on it. I'm just saying that releasing and re-acquiring the GIL around fast syscalls is probaly not a good idea. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
Antoine Pitrou pit...@free.fr added the comment: Just to be clear, I'm not at all criticizing the current GIL implementation, there's been a great work done on it. I'm just saying that releasing and re-acquiring the GIL around fast syscalls is probaly not a good idea. If these syscalls aren't likely to yield control to another thread, then I agree there's no point in releasing the GIL around them. (but is it the case that they are always fast? for example, how about dup() on a network file system? or is it indifferent?) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
Charles-Francois Natali neolo...@free.fr added the comment: 2011/3/3 Antoine Pitrou rep...@bugs.python.org: Antoine Pitrou pit...@free.fr added the comment: Just to be clear, I'm not at all criticizing the current GIL implementation, there's been a great work done on it. I'm just saying that releasing and re-acquiring the GIL around fast syscalls is probaly not a good idea. If these syscalls aren't likely to yield control to another thread, then I agree there's no point in releasing the GIL around them. (but is it the case that they are always fast? for example, how about dup() on a network file system? or is it indifferent?) The initial open can take long, but once it's open, calling dup just implies copying a reference to the open file (a pointer) to the file descriptor table. No I/O is done (I tested it one a NFS mount). Now, I don't know Windows at all, but I'm pretty sure that every operating system does more or less the same thing, and that those three calls (there might be others) don't block. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11382] some posix module functions unnecessarily release the GIL
Charles-Francois Natali neolo...@free.fr added the comment: Attached is a patch removing useless calls to Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS for several posix functions. It's straigthforward, but since I only have Linux boxes, I couldn't test it under Windows. -- keywords: +patch Added file: http://bugs.python.org/file20988/gil_release_py3k.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11382 ___Index: Modules/posixmodule.c === --- Modules/posixmodule.c (révision 88734) +++ Modules/posixmodule.c (copie de travail) @@ -3041,9 +3041,7 @@ if (!PyArg_ParseTuple(args, ii, which, who)) return NULL; errno = 0; -Py_BEGIN_ALLOW_THREADS retval = getpriority(which, who); -Py_END_ALLOW_THREADS if (errno != 0) return posix_error(); return PyLong_FromLong((long)retval); @@ -3063,9 +3061,7 @@ if (!PyArg_ParseTuple(args, iii, which, who, prio)) return NULL; -Py_BEGIN_ALLOW_THREADS retval = setpriority(which, who, prio); -Py_END_ALLOW_THREADS if (retval == -1) return posix_error(); Py_RETURN_NONE; @@ -5712,9 +5708,7 @@ return NULL; if (!_PyVerify_fd(fd)) return posix_error(); -Py_BEGIN_ALLOW_THREADS fd = dup(fd); -Py_END_ALLOW_THREADS if (fd 0) return posix_error(); return PyLong_FromLong((long)fd); @@ -5733,9 +5727,7 @@ return NULL; if (!_PyVerify_fd_dup2(fd, fd2)) return posix_error(); -Py_BEGIN_ALLOW_THREADS res = dup2(fd, fd2); -Py_END_ALLOW_THREADS if (res 0) return posix_error(); Py_INCREF(Py_None); @@ -6116,9 +6108,7 @@ HFILE read, write; APIRET rc; -Py_BEGIN_ALLOW_THREADS rc = DosCreatePipe( read, write, 4096); -Py_END_ALLOW_THREADS if (rc != NO_ERROR) return os2_error(rc); @@ -6127,9 +6117,7 @@ #if !defined(MS_WINDOWS) int fds[2]; int res; -Py_BEGIN_ALLOW_THREADS res = pipe(fds); -Py_END_ALLOW_THREADS if (res != 0) return posix_error(); return Py_BuildValue((ii), fds[0], fds[1]); @@ -6137,9 +6125,7 @@ HANDLE read, write; int read_fd, write_fd; BOOL ok; -Py_BEGIN_ALLOW_THREADS ok = CreatePipe(read, write, NULL, 0); -Py_END_ALLOW_THREADS if (!ok) return win32_error(CreatePipe, NULL); read_fd = _open_osfhandle((Py_intptr_t)read, 0); ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
