[issue9260] A finer grained import lock
Changes by Antoine Pitrou pit...@free.fr: -- status: pending - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset edb9ce3a6c2e by Antoine Pitrou in branch 'default': Issue #9260: A finer-grained import lock. http://hg.python.org/cpython/rev/edb9ce3a6c2e -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: I have now pushed the patch. -- resolution: - fixed status: open - pending ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by Antoine Pitrou pit...@free.fr: -- stage: patch review - committed/rejected status: pending - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by Antoine Pitrou pit...@free.fr: -- status: open - pending ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: Does anyone else want to review this patch? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Brett Cannon br...@python.org added the comment: I don't feel the need to, but I can in a few days if you want me to (just let me know if you do). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: I had forgotten to tackle threadless builds, this patch fixes it. -- Added file: http://bugs.python.org/file25521/module_locks9.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by Andrew Svetlov andrew.svet...@gmail.com: -- nosy: +asvetlov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: Updated patch against tip. I also changed the internal API of module locks a bit (acquire() raises _DeadlockError instead of returning False, and deadlock detection is not optional anymore). -- Added file: http://bugs.python.org/file25496/module_locks8.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: Updated patch also makes PyImport_ImportModuleNoBlock a simple alias of PyImport_ImportModule. -- Added file: http://bugs.python.org/file25466/module_locks4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by Antoine Pitrou pit...@free.fr: Removed file: http://bugs.python.org/file25466/module_locks4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by Antoine Pitrou pit...@free.fr: Added file: http://bugs.python.org/file25469/module_locks4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by Eric Snow ericsnowcurren...@gmail.com: -- nosy: +eric.snow ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: Updated patch also adds unit tests for the module locks and the deadlock avoidance algorithm. -- Added file: http://bugs.python.org/file25470/module_locks5.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: Updated patch with a couple new tests. -- stage: - patch review Added file: http://bugs.python.org/file25471/module_locks6.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Martin v. Löwis mar...@v.loewis.de added the comment: I still wonder whether Graham Dumpleton's observation has merits. Suppose we have these modules # a.py time.sleep(10) import b # b.py time.sleep(10) import a # main.py def x(): import a def y(): import b Now, if x and y are executed in separate threads - won't it deadlock? -- nosy: +loewis ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: Now, if x and y are executed in separate threads - won't it deadlock? Well, the patch has a deadlock avoidance mechanism, and it includes unit tests for precisely this situation. I cannot promise the algorithm is perfect (although there *are* a bunch of tests), but it looks correct from here. :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Martin v. Löwis mar...@v.loewis.de added the comment: Can you please elaborate in the patch what the deadlock avoidance does? AFAICT, the comment explains that it is able to detect deadlocks, but nowhere says what it does when it has detected a deadlock. Also, please submit patches against default's head, or stop using git-style diffs, to enable Rietveld review. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: Updated patch against tip, and with a comment of what deadlock avoidance does (in _ModuleLock.acquire's docstring). -- Added file: http://bugs.python.org/file25475/module_locks7.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Martin v. Löwis mar...@v.loewis.de added the comment: The patch parser of Rietveld actually choked on the git binary diff. It now skips over these chunks. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: Ok, here is a draft patch for the new importlib. Several issues with this patch: - introduces a pure Python function (_lock_unlock_module) on the fast import path - synchronization issues due to interruptibility of pure Python code (see _ModuleLock.acquire) - afterfork fix-up necessary - relies on _thread.RLock for bootstrapping reasons - module locks are immortal -- Added file: http://bugs.python.org/file25392/module_locks.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: New patch gets rid of the reliance on _thread.RLock (uses non-recursive locks instead), and should solve the synchronization issue. Other issues remain. -- Added file: http://bugs.python.org/file25394/module_locks2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: Updated patch fixes the performance issue and disposes of module locks when they aren't used anymore. Only the afterfork question remains. Should I hook in threading's own facility? Should we wait for an atfork module? Something else. -- Added file: http://bugs.python.org/file25398/module_locks3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by Antoine Pitrou pit...@free.fr: Removed file: http://bugs.python.org/file25398/module_locks3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by Antoine Pitrou pit...@free.fr: Added file: http://bugs.python.org/file25399/module_locks3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Charles-François Natali neolo...@free.fr added the comment: That's true. Do you think temptatively acquiring the lock (without blocking) would solve the issue? I think it should work. Something along those lines: while True: if lock.acquire(0): lock.tstate = tstate return True else: if detect_circularity(): return False global_lock.release() saved = save_tstate() yield() restore_tstate(saved) global_lock.acquire() However, I find this whole mechanism somewhat complicated, so the question really is: what are we trying to solve? If we just wan't to avoid deadlocks, a trylock with the global import lock will do the trick. If, on the other hand, we really want to reduce the number of cases where a deadlock would occur by increasing the locking granularity, then it's the way to go. But I'm not sure it's worth the extra complexity (increasing the locking granularity is usually a proven recipe to introduce deadlocks). Isn't this limit only about named semaphores? Or does it apply to anonymous semaphores as well? I'm no FreeBSD expert, but AFAICT, POSIX SEM_NSEMS_MAX limit doesn't seem to make a distinction between named and anonymous semaphores. From POSIX sem_init() man page: [ENOSPC] A resource required to initialise the semaphore has been exhausted, or the limit on semaphores (SEM_NSEMS_MAX) has been reached. Also, a quick search returned those links: http://ftp.es.freebsd.org/pub/FreeBSD/development/FreeBSD-CVS/src/sys/sys/ksem.h,v http://translate.google.fr/translate?hl=frsl=rutl=enu=http%3A%2F%2Fforum.nginx.org%2Fread.php%3F21%2C202865%2C202865 So it seems that sem_init() can fail when the max number of semaphores is reached. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: If, on the other hand, we really want to reduce the number of cases where a deadlock would occur by increasing the locking granularity, then it's the way to go. Yes, that's the point. Today you have to be careful when mixing imports and threads. The problems is that imports can be implicit, inside a library call for example (and putting imports inside functions is a way of avoiding circular imports, or can also allow to reduce startup time). Some C functions try to circumvent the problem by calling PyImport_ModuleNoBlock, which is ugly and fragile in its own way (it will fail if the import lock is taken, even if there wouldn't be a deadlock: a very primitive kind of deadlock avoidance indeed). Also, a quick search returned those links: http://ftp.es.freebsd.org/pub/FreeBSD/development/FreeBSD-CVS/src/sys/sys/ksem.h,v http://translate.google.fr/translate?hl=frsl=rutl=enu=http%3A%2F%2Fforum.nginx.org%2Fread.php%3F21%2C202865%2C202865 So it seems that sem_init() can fail when the max number of semaphores is reached. As they say, Kritirii choice of the number 30 in the XXI century is unclear. :-) File objects also have a per-object lock, so I guess opening 30 files under FreeBSD would fail. Perhaps we need to fallback on the other lock implementation on certain platforms? (our FreeBSD 8.2 buildbot looks pretty much stable, was the number of semaphores tweaked on it?) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: I believe this new patch should be much more robust than the previous one. It also adds a test for the improvement (it freezes an unpatched interpreter). -- Added file: http://bugs.python.org/file24114/implock5.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: It owns the lock, but hasn't yet updated the lock's owner (lock-tstate), so another thread calling detect_circularity() will think that this lock is available, and will proceed, which can eventually lead to a deadlock. That's true. Do you think temptatively acquiring the lock (without blocking) would solve the issue? Also, I think that locks will use POSIX semaphores on systems that support only a limited number of them (such as FreeBSD 7), and this might fail in case of nested imports (the infamous ENFILE). I'd have to double check this, though. Isn't this limit only about named semaphores? Or does it apply to anonymous semaphores as well? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Charles-François Natali neolo...@free.fr added the comment: IIUC, the deadlock avoidance code just checks that acquiring a per-module lock won't create a cycle. However, I think there's a race, because the cycle detection and the lock acquisition is not atomic. For example, let's say we have a thread exactly here in in acquire_import_lock(): PyThread_acquire_lock(lock-lock, 1); /* thread inside PyEval_RestoreThread(), waiting for the GIL */ PyEval_RestoreThread(saved); lock-waiters--; } assert(lock-level == 0); lock-tstate = tstate; It owns the lock, but hasn't yet updated the lock's owner (lock-tstate), so another thread calling detect_circularity() will think that this lock is available, and will proceed, which can eventually lead to a deadlock. Also, I think that locks will use POSIX semaphores on systems that support only a limited number of them (such as FreeBSD 7), and this might fail in case of nested imports (the infamous ENFILE). I'd have to double check this, though. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: New prototype with per-module import locks and deadlock avoidance. When a deadlock due to threaded circular imports is detected, the offending import returns the partially constructed module object (as would happen in single-threaded mode). Probably lacks a test and some cleanup. -- nosy: +neologix versions: +Python 3.3 -Python 3.2 Added file: http://bugs.python.org/file24101/implock3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by STINNER Victor victor.stin...@haypocalc.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: Graham Dumpleton graham.dumple...@gmail.com added the comment: How is this going to deal with cyclical imports where different threads could import at the same time different modules within that cycle? I need to look through the proposed patch and work out exactly what it does, but am concerned about whether this approach would cause the classic deadlock problem if not done right? You're right, I hadn't thought about that. Additional machinery will be needed to detect potential deadlocks (and break them). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: That's my point; loaders are using the lock implicitly so that's why we don't need to worry about the global import lock just for path hooks. It seems like you are suggesting using the global import lock purely for compatibility, and what I am saying is that loaders don't care so there is no compatibility to care about. I say only use the global import lock for serializing creation. What is your take on the threadimp2.patch in issue9251? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Brett Cannon br...@python.org added the comment: I'll have a look when I can (hopefully EuroPython). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
New submission from Antoine Pitrou pit...@free.fr: This is an implementation of the idea suggested in: http://mail.python.org/pipermail/python-dev/2003-February/033445.html The patch creates a dictionary of reentrant locks keyed by module full name. Trying to import a module or package will first get the lock for that module (or, if necessary, create it) and then acquire it. This is done for any module type. The global import lock is still there, but only used for two things: - serializing first time creation of module-specific locks - protection of imports based on import hooks, since we don't know whether third-party import hooks are themselves thread-safe Semantics of the public C API are unchanged, because it is not clear whether they should be or not (concerns of usefulness vs. compatibility). For example, PyImport_ImportModuleNoBlock() still uses the global import lock but this could be relaxed in a later patch. -- components: Interpreter Core files: implock.patch keywords: patch messages: 110287 nosy: brett.cannon, christian.heimes, grahamd, gvanrossum, ncoghlan, pitrou priority: normal severity: normal status: open title: A finer grained import lock type: feature request versions: Python 3.2 Added file: http://bugs.python.org/file17998/implock.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by Antoine Pitrou pit...@free.fr: Removed file: http://bugs.python.org/file17998/implock.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by Antoine Pitrou pit...@free.fr: Added file: http://bugs.python.org/file17999/implock.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by Alexander Belopolsky belopol...@users.sourceforge.net: -- nosy: +belopolsky ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Brett Cannon br...@python.org added the comment: So I say we don't worry about loaders being thread-safe. If __import__ handles the locking for a specific module then it will hold the lock on behalf of the loader. Now if someone decides to call load_module on their own, that's there business, but they should be aware of what could happen if they do that without acquiring the lock themselves. Otherwise we just make sure to provide a context manager that takes the name of the module and people can use that when they make their call to loader.load_module. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: So I say we don't worry about loaders being thread-safe. If __import__ handles the locking for a specific module then it will hold the lock on behalf of the loader. Yes but what happens if two different modules are imported from two different threads, and handled by the same loader? The loader could have global structures which rely on serialization of imports for consistency. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Brett Cannon br...@python.org added the comment: On Wed, Jul 14, 2010 at 12:34, Antoine Pitrou rep...@bugs.python.orgwrote: Antoine Pitrou pit...@free.fr added the comment: So I say we don't worry about loaders being thread-safe. If __import__ handles the locking for a specific module then it will hold the lock on behalf of the loader. Yes but what happens if two different modules are imported from two different threads, and handled by the same loader? The loader could have global structures which rely on serialization of imports for consistency. That's why I said we should supply a context decorator (or function) which will handle the lock appropriately, taking the name of the module to import as an argument so the locking is fine-grained. -- Added file: http://bugs.python.org/file18005/unnamed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___brbrdiv class=gmail_quoteOn Wed, Jul 14, 2010 at 12:34, Antoine Pitrou span dir=ltrlt;a href=mailto:rep...@bugs.python.org;rep...@bugs.python.org/agt;/span wrote:brblockquote class=gmail_quote style=margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex; br Antoine Pitrou lt;a href=mailto:pit...@free.fr;pit...@free.fr/agt; added the comment:br div class=imbr gt; So I say we don#39;t worry about loaders being thread-safe. If __import__br gt; handles the locking for a specific module then it will hold the lockbr gt; on behalf of the loader.br br /divYes but what happens if two different modules are imported from twobr different threads, and handled by the same loader? The loader could havebr global structures which rely on serialization of imports forbr consistency.br/blockquotedivbr/divdivThat#39;s why I said we should supply a context decorator (or function) which will handle the lock appropriately, taking the name of the module to import as an argument so the locking is fine-grained./div /div ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: That's why I said we should supply a context decorator (or function) which will handle the lock appropriately, taking the name of the module to import as an argument so the locking is fine-grained. Ok, so what are you saying is that we can break compatibility for non thread-safe import loaders which currently work fine? (I have nothing against it, just trying to be sure we agree on the implications) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Brett Cannon br...@python.org added the comment: What I'm saying is that loaders are quite possibly not thread-safe already, so we don't need to do any special for them. If you look at PEP 302 you will notice not a single mention of loaders needing to care about the import lock because there is no mention of the import lock! So changing the locking mechanism most likely won't break loaders because they are not using the current import lock anyway and so already have their own issues. As long as __import__ does the proper locking on behalf of loaders and we provide a way for people to use the lock if they want to then I am not worried about the impact on loaders. For example, this will change the logic in importlib where the current import lock is grabbed, but otherwise won't change a thing in terms of the code for the various loaders it implements. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Antoine Pitrou pit...@free.fr added the comment: So changing the locking mechanism most likely won't break loaders because they are not using the current import lock anyway and so already have their own issues. Are you sure they aren't using it implicitly? In vanilla py3k, PyImport_ImportModuleLevel() takes the import lock therefore it protects any inner code, including the various hooks. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Changes by Alexander Belopolsky belopol...@users.sourceforge.net: Removed file: http://bugs.python.org/file18005/unnamed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Graham Dumpleton graham.dumple...@gmail.com added the comment: How is this going to deal with cyclical imports where different threads could import at the same time different modules within that cycle? I need to look through the proposed patch and work out exactly what it does, but am concerned about whether this approach would cause the classic deadlock problem if not done right? FWIW, this concept of a lock per module is what I used in the mod_python module importer when it was rewritten. I would have to go look back over that code and see how the way the concept is being implemented differs, but there was one remaining potential race condition in the mod_python code which could in rare instances cause a problem. I never did get around to fixing it. Anyway, what I did learn was that this approach isn't necessarily as simple as it may seem so it will need some really good analysis on whatever solution is developed to ensure subtle problems don't come up. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9260] A finer grained import lock
Brett Cannon br...@python.org added the comment: That's my point; loaders are using the lock implicitly so that's why we don't need to worry about the global import lock just for path hooks. It seems like you are suggesting using the global import lock purely for compatibility, and what I am saying is that loaders don't care so there is no compatibility to care about. I say only use the global import lock for serializing creation. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9260 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
