[Python-Dev] Dealing with import lock deadlock in Import Hooks
[I initially posted this email to python-list but didn't get any reply, probably because this is too related to python core, so I'm posting it again here, hope that's ok...] Hello, I'm currently working on implementing Import Hooks (PEP302) with Python 2.7 to be able to import modules whose code is in ZODB. However, I have stumbled upon a widely known issue about import deadlock[0][1] (note that this issue is not directly related to ZODB, but a more general question about dealing with import lock deadlock for Import Hooks), basically: Thread 1 is trying to import a module 'foo.bar' (where 'foo' is a package containing dynamic modules) handled by Import Hooks I implemented, so import lock is acquired before even running the hooks (Python/import.c:PyImport_ImportModuleLevel()). Then, these import hooks try to load objects from ZODB and a request is sent and handled by another thread (Thread 2) which itself tries to import another module. Of course, this causes a deadlock because the first thread still holds import lock. I have thought about the following solutions: * Backport the patch applied in python 3.3 from issue 9260[0]. This would be the best option because it would mean that even when trying to import any module from package 'foo', other modules and packages can be imported, which would solve my issue. However, I'm not sure it could be released into python 2.7? * Within the hooks, protect the Import Hooks with a separate lock for the loader method. This would prevent any other thread to import any modules from 'foo' package but still allows to call the finder method (ignoring module fullname not starting with 'foo') along with other finder methods, so that other ZODB modules can be imported. Then, in the loader method, until the module is actually inserted into sys.modules and then other load_module() PEP302 responsabilities being taken care of (such as exec the code), release the import lock so that Thread 2 can process requests and send objects back to Thread 1. About the finder method, I think that the separate lock is enough and releasing the import lock until the end of the method should be enough. However, even after trying to understand import.c, I'm not sure this is enough and that releasing import lock would not have nasty side-effects, any thoughts about that? * Fix the ZODB code to not avoid import but to me this seems like a dirty hack because it could happen again and I would prefer to fix this issue once and for all. Any thoughts or suggestion welcome, thanks! Regards, Arnaud Fontaine ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks
>I'm currently working on implementing Import Hooks (PEP302) with Python > 2.7 to be able to import modules whose code is in ZODB. However, I have > stumbled upon a widely known issue about import deadlock[0][1] (...) In Python 3.3, the import machinery has been rewritten (importlib is used by default) and the import lock is now per module, no more global. Backporting such huge change is difficult and risky. Upgrading to Python 3.3 is more future proof and don't require to hack Python 2.7. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks
Hi Arnaud, On Mon, Aug 12, 2013 at 9:39 AM, Arnaud Fontaine wrote: > Thread 1 is trying to import a module 'foo.bar' (where 'foo' is a > package containing dynamic modules) handled by Import Hooks I > implemented, so import lock is acquired before even running the hooks > (Python/import.c:PyImport_ImportModuleLevel()). Then, these import > hooks try to load objects from ZODB and a request is sent and handled > by another thread (Thread 2) which itself tries to import another > module. A quick hack might be to call imp.release_lock() and imp.acquire_lock() explicitly, from your import hook code, around calls to ZODB. A bientôt, Armin. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks
Victor Stinner writes: >>I'm currently working on implementing Import Hooks (PEP302) with Python >> 2.7 to be able to import modules whose code is in ZODB. However, I have >> stumbled upon a widely known issue about import deadlock[0][1] (...) > > In Python 3.3, the import machinery has been rewritten (importlib is used > by default) and the import lock is now per module, no more global. Yes, I saw the bug report and its patch implementing the import lock per module (mentioned in my initial email) and watched the presentation by Brett Cannon (BTW, I could not find the diagram explained during the presentation, anyone knows if it's available somewhere?). > Backporting such huge change is difficult and risky. > > Upgrading to Python 3.3 is more future proof and don't require to hack > Python 2.7. I wish I could use Python 3.3 but unfortunately, Zope 2 does not support it. What about the other solution I suggested though? Regards, -- Arnaud Fontaine ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] NULL allowed in PyErr_SetString and friends?
Hello, It seems NULL is allowed as the first argument of PyErr_Format, PyErr_SetString and PyErr_SetObject. Moreover, it means "clear the error indicator". However, this is not mentioned in the docs. I was wondering if we should officialize this behaviour or change it. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] NULL allowed in PyErr_SetString and friends?
On Mon, Aug 12, 2013 at 5:10 AM, Antoine Pitrou wrote:
>
> Hello,
>
> It seems NULL is allowed as the first argument of PyErr_Format,
> PyErr_SetString and PyErr_SetObject. Moreover, it means "clear the
> error indicator". However, this is not mentioned in the docs. I was
> wondering if we should officialize this behaviour or change it.
>
Since the same capability is available much more clearly through
PyError_Clear (and also through PyError_Restore(NULL, NULL, NULL)), IMHO we
should at least:
1. Document that NULL is not allowed in PyErr_Set{Object|String}
2. Switch all actual uses of that idiom in the stdlib to PyError_Clear
If we don't fear external code breakaga that relies on this undocumented
behavior, we can also add explicit treating of NULL in
PyErr_Set{Object|String} (maybe even asserting). Otherwise, we can just
keep the behavior as is for now, though make it more correct: to do the
reset it would call PyError_Restore(NULL, value, tb) even though
PyError_Restore documents that all args should be NULL to actually clear
the indicator.
Eli
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks
On Mon, Aug 12, 2013 at 5:12 AM, Arnaud Fontaine wrote: > Victor Stinner writes: > > >>I'm currently working on implementing Import Hooks (PEP302) with Python > >> 2.7 to be able to import modules whose code is in ZODB. However, I have > >> stumbled upon a widely known issue about import deadlock[0][1] (...) > > > > In Python 3.3, the import machinery has been rewritten (importlib is used > > by default) and the import lock is now per module, no more global. > > Yes, I saw the bug report and its patch implementing the import lock per > module (mentioned in my initial email) and watched the presentation by > Brett Cannon (BTW, I could not find the diagram explained during the > presentation, anyone knows if it's available somewhere?). > http://prezi.com/mqptpza9xbic/?utm_campaign=share&utm_medium=copy -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] SSL issues in Python stdlib and 3rd party code
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hello, last week Ryan Sleevi of the Google Chrome Security Team has informed us about about two issues in Python's SSL module. I already new about the cause of the first bug and suspected that our SSL module suffers from the second bug but I was unable to prove it. Both issues are security issues but their impact is limited if you trust only trustworthy root certification authorities. Any decent root CA would should not sign a malicious cert with NULL bytes in a subjectAltName dNSName field or with wildcards like *.*.com. By the way if you are using the cacert.pem from curl, please update your bundle ASAP. I have found a bug in its Mozilla certdata parser, too. bug #1: ssl.match_hostname() wildcard matching - -- ssl.match_hostname() doesn't implement RFC 6125 wildcard matching rules. Affected versions: - - Python 3.2 (< 3.2.5) - - Python 3.3 (< 3.3.3) - - Python 3.4a1 - - requests < 1.2.3 https://pypi.python.org/pypi/requests - - backports.ssl_match_hostname (<3.2a3) https://pypi.python.org/pypi/backports.ssl_match_hostname/ - - urllib3 < 1.6 https://github.com/shazow/urllib3 Bug reports: http://bugs.python.org/issue17997 https://github.com/kennethreitz/requests/issues/1528 https://bitbucket.org/brandon/backports.ssl_match_hostname/issue/2/match_hostname-doesnt-implement-rfc-6125 Patch: http://bugs.python.org/issue17997 has a preliminary patch. The handling of IDN A-labels is still a bit controversial, though. bug #2 failure to handle NULL bytes in subjectAltName - - It's basically the same issue as CVE-2013-4073. Python uses GENERAL_NAME_print() to turn a GERNAL_NAME entry into a C string. But GENERAL_NAME_print() doesn't handle embedded NULL bytes in ASN1_STRINGs correctly. You can read more about the issue at http://www.ruby-lang.org/en/news/2013/06/27/hostname-check-bypassing-vulnerability-in-openssl-client-cve-2013-4073/ Affected versions: - - Python 2.6 (< 2.6.8) - - Python 2.7 (< 2.7.5) - - Python 3.2 (< 3.2.5) - - Python 3.3 (< 3.3.3) - - Python 3.4a1 - - PyOpenSSL < 0.13 https://pypi.python.org/pypi/pyOpenSSL - - eGenix.com pyOpenSSL Distribution with PyOpenSSL < 0.13 https://pypi.python.org/pypi/M2Crypto - - M2Crypto < 0.21.1 http://www.egenix.com/products/python/pyOpenSSL/ Bug report: http://bugs.python.org/issue18709 Patches: http://bugs.python.org/issue18709 has patches for 2.7, 3.3 and default https://code.launchpad.net/~heimes/pyopenssl/pyopenssl/+merge/179673 Jean-Paul Calderone is going to release 0.13.1 soonish. It's going to contain just my fix for the issue. Marc-Andre Lemburg will build a new version of eGenix.com pyOpenSSL Distribution shortly after. I'm not going to work on a patch for M2Crypto as I don't understand SWIG. I have contacted Heikki Toivonen for M2Crypto but haven't heard back from him yet. related issue: Mozilla's certdata.txt and CKT_NSS_MUST_VERIFY_TRUST - --- Recently I found bugs in curl's mk-ca-bundle.pl script, its cacert.pem and in the CA bundle of eGenix.com pyOpenSSL Distribution. Both failed to handle a new option in Mozilla's certdata.txt database correctly. As a consequence the root CA bundles contained additionally and untrustworthy root certificates. I'm not sure about the severity of the issue. Curl has already fixed its script week ago. Marc-Andre Lemburg is going to release a new distribution very soon. https://github.com/bagder/curl/commit/51f0b798fa http://curl.haxx.se/docs/caextract.html Background information: https://www.imperialviolet.org/2012/01/30/mozillaroots.html http://lists.debian.org/debian-release/2012/11/msg00411.html http://p11-glue.freedesktop.org/doc/storing-trust-policy/storing-trust-existing.html I like to thank Ryan Sleevi (Google), Chris Palmer (Google), Marc-Andre Lemburg (eGenix.com, Python core dev), Jean-Paul Calderone (PyOpenSSL), Antoine Pitrou (Python core dev), Daniel Stenberg (curl), Günter Knauf (curl) and everybody else who was involved in reporting and fixing these issues. Regards, Christian -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQIcBAEBCgAGBQJSCRjUAAoJEMeIxMHUVQ1F81UP/R8RqAFR3cODI4SgFzmvOSXa u5OHfdQcgVfQbs5+oeZgqBu5aQ2W4SHKd/wSqCCrB1CbsZDZu9Y5LRVo/MgKMuNs /c49FAyNKGUPc04RtBPfjSQ8GWlk4ziayW9PAuziBGD+ExMlSzXk1CwRwPJVPMkA 8xV48QThT4U5L1WRS4AH3XdgIpPWwJswNCuw9KjlIP/b1LZIrVvUg9rb4azVf0qu Am9IlCwIb2sMNQU1s+sADht6B3Ka4tC8ej8VoWHnTEh8T8RJgcG3j3P2GFPx0YzD 35ISU6k/Dg/dEIJjawI7Uk+dXqxhMfCWFz5Yoy7TaUtYTFAFISqys88+I/H3PxNV mewUNdRFO9ej2vyikI8s1FwGVaORYEIVtkOKzLRa7mc5ZEdh6MjZ2l3bH2GszO+J mRzDwQjipnp8NwOjn9eipdKNaywFoGnDmoydxf3w9Qq6MPsdSiqtBHPWRP5K+091 rM+49v0MAenvxtkb8IJsBbSzVMs66uwfRYl1KYyvXNpGb4TSH/GlrUqRqaX97NW2 x1NprclxVWri5/kWnV4YvqJ6OmDcigCVI780+rQFSoqMk4JKDUgUsl451KvB4ATL 5OfZ9VaVhXu6Ydrjb3bRZHuK
Re: [Python-Dev] SSL issues in Python stdlib and 3rd party code
Hi, On Mon, 12 Aug 2013 19:18:17 +0200 Christian Heimes wrote: > related issue: Mozilla's certdata.txt and CKT_NSS_MUST_VERIFY_TRUST > - --- > > Recently I found bugs in curl's mk-ca-bundle.pl script, its cacert.pem > and in the CA bundle of eGenix.com pyOpenSSL Distribution. Both failed > to handle a new option in Mozilla's certdata.txt database correctly. > As a consequence the root CA bundles contained additionally and > untrustworthy root certificates. I'm not sure about the severity of > the issue. Which goes to show that not bundling our own set of CA certificates is the safest route. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Deprecating the formatter module
At the PyCon CA sprint someone discovered the formatter module had somewhat low code coverage. We discovered this is because it's tested by test_sundry, i.e. it's tested by importing it and that's it. We then realized that it isn't really used by anyone (pydoc uses it but it should have been using textwrap). Looking at the history of the module it has just been a magnet for cleanup revisions and not actual usage or development since Guido added it back in 1995. I have created http://bugs.python.org/issue18716 to deprecate the formatter module for removal in Python 3.6 unless someone convinces me otherwise that deprecation and removal is the wrong move. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecating the formatter module
On 12 August 2013 20:22, Brett Cannon wrote: > At the PyCon CA sprint someone discovered the formatter module had > somewhat low code coverage. We discovered this is because it's tested by > test_sundry, i.e. it's tested by importing it and that's it. > > We then realized that it isn't really used by anyone (pydoc uses it but it > should have been using textwrap). Looking at the history of the module it > has just been a magnet for cleanup revisions and not actual usage or > development since Guido added it back in 1995. > > I have created http://bugs.python.org/issue18716 to deprecate the > formatter module for removal in Python 3.6 unless someone convinces me > otherwise that deprecation and removal is the wrong move. > I can see no reason to object. But having looked at the module for the first time on the basis of this email, I have to say that if I'd stumbled across it by chance, my reaction would have been that it was another one of Python's "hidden gems" that I'd never been aware of. I would then, of course, have filed it for future reference "should I need it", and never actually used it. So I'm OK for it to go, but a little sad nevertheless :-) Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecating the formatter module
On 08/12/2013 04:11 PM, Paul Moore wrote: [...] if I'd stumbled across it by chance, my reaction would have been that it was another one of Python's "hidden gems" that I'd never been aware of. Hidden "gem"? No. Hidden "paste diamond", maybe. YAGNI, //arry/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecating the formatter module
On Mon, Aug 12, 2013 at 12:22 PM, Brett Cannon wrote: > At the PyCon CA sprint someone discovered the formatter module had > somewhat low code coverage. We discovered this is because it's tested by > test_sundry, i.e. it's tested by importing it and that's it. > > We then realized that it isn't really used by anyone (pydoc uses it but it > should have been using textwrap). Looking at the history of the module it > has just been a magnet for cleanup revisions and not actual usage or > development since Guido added it back in 1995. > > I have created http://bugs.python.org/issue18716 to deprecate the > formatter module for removal in Python 3.6 unless someone convinces me > otherwise that deprecation and removal is the wrong move. > I wish we had a way to collect real-world usage on such things. I tried a couple of code search engines, but this one is difficult to unravel because many Python packages have their own formatter module (for example Django, pygments) that probably does something different. Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (merge 2.7 -> 2.7): Clean merge
Hi, On Mon, Aug 12, 2013 at 10:51 PM, david.wolever wrote: > http://hg.python.org/cpython/rev/0f4d971b0cee > changeset: 85138:0f4d971b0cee > branch: 2.7 > parent: 85137:102b3e257dca > parent: 83899:ef037ad304c1 > user:David Wolever > date:Thu May 23 17:51:58 2013 -0400 > summary: > Clean merge > > files: > .hgtags|1 + > Doc/c-api/exceptions.rst | 38 +- > Doc/c-api/intro.rst|4 +- > Doc/faq/design.rst |4 +- > Doc/faq/programming.rst| 86 + > Doc/glossary.rst |8 + > Doc/howto/advocacy.rst | 355 --- > Doc/howto/index.rst|1 - > Doc/howto/sockets.rst |8 +- > Doc/howto/urllib2.rst | 12 +- > Doc/library/codecs.rst | 172 ++- > Doc/library/collections.rst|4 +- > Doc/library/compileall.rst |2 +- > Doc/library/ctypes.rst |2 +- > Doc/library/io.rst |3 + > Doc/library/itertools.rst |4 +- > Doc/library/numbers.rst|8 +- > Doc/library/operator.rst | 47 +- > Doc/library/resource.rst | 21 +- > Doc/library/socket.rst | 16 +- > Doc/library/ssl.rst| 16 +- > Doc/library/stdtypes.rst | 28 +- > Doc/library/string.rst |5 +- > Doc/library/unittest.rst |2 + > Doc/library/urllib.rst |7 + > Doc/library/urllib2.rst| 15 +- > Doc/reference/datamodel.rst|9 +- > Doc/reference/expressions.rst | 15 +- > Doc/reference/simple_stmts.rst |3 + > Doc/tutorial/inputoutput.rst | 23 +- > Doc/tutorial/modules.rst |7 +- > Doc/using/mac.rst | 14 +- > Include/object.h | 16 +- > Include/patchlevel.h |4 +- > Lib/_weakrefset.py |6 + > Lib/collections.py |2 - > Lib/ctypes/test/__init__.py|2 +- > Lib/ctypes/test/test_wintypes.py | 43 + > Lib/ctypes/util.py |2 +- > Lib/distutils/__init__.py |2 +- > Lib/filecmp.py |2 +- > Lib/gzip.py| 69 +- > Lib/idlelib/Bindings.py|4 + > Lib/idlelib/EditorWindow.py| 31 +- > Lib/idlelib/PyShell.py |1 - > Lib/idlelib/help.txt |3 +- > Lib/idlelib/idlever.py |2 +- > Lib/idlelib/run.py |5 + > Lib/logging/handlers.py| 36 +- > Lib/mimetypes.py |2 + > Lib/multiprocessing/pool.py|2 + > Lib/multiprocessing/synchronize.py |2 +- > Lib/multiprocessing/util.py|5 +- > Lib/pickle.py |2 +- > Lib/plistlib.py|4 +- > Lib/pydoc_data/topics.py | 18 +- > Lib/sre_parse.py |6 +- > Lib/ssl.py | 26 +- > Lib/tarfile.py | 12 +- > Lib/test/pickletester.py |2 + > Lib/test/test_base64.py| 26 + > Lib/test/test_bz2.py | 31 +- > Lib/test/test_collections.py |2 +- > Lib/test/test_dictviews.py |5 + > Lib/test/test_gdb.py | 46 +- > Lib/test/test_gzip.py | 17 - > Lib/test/test_io.py|4 +- > Lib/test/test_mimetypes.py |2 + > Lib/test/test_multiprocessing.py | 32 +- > Lib/test/test_plistlib.py | 12 + > Lib/test/test_pydoc.py | 57 +- > Lib/test/test_re.py| 11 + > Lib/test/test_sax.py | 20 + > Lib/test/test_support.py |9 + > Lib/test/test_tarfile.py |8 + > Lib/test/test_tcl.py | 18 +- > Lib/test/test_weakset.py |6 + > Lib/test/test_winreg.py| 12 +- > Lib/test/test_zipfile.py | 10 +- > Lib/test/testbz2_bigmem.bz2| Bin > Lib/threading.py | 42 +- > Lib/xml/sax/saxutils.py|8 +- > M
Re: [Python-Dev] Deprecating the formatter module
On Mon, 12 Aug 2013 14:22:01 -0700 Eli Bendersky wrote: > On Mon, Aug 12, 2013 at 12:22 PM, Brett Cannon wrote: > > > At the PyCon CA sprint someone discovered the formatter module had > > somewhat low code coverage. We discovered this is because it's tested by > > test_sundry, i.e. it's tested by importing it and that's it. > > > > We then realized that it isn't really used by anyone (pydoc uses it but it > > should have been using textwrap). Looking at the history of the module it > > has just been a magnet for cleanup revisions and not actual usage or > > development since Guido added it back in 1995. > > > > I have created http://bugs.python.org/issue18716 to deprecate the > > formatter module for removal in Python 3.6 unless someone convinces me > > otherwise that deprecation and removal is the wrong move. > > > > I wish we had a way to collect real-world usage on such things. I tried a > couple of code search engines, but this one is difficult to unravel because > many Python packages have their own formatter module (for example Django, > pygments) that probably does something different. "Ohloh code search" shows a couple matches for AbstractFormatter in Python projects: http://code.ohloh.net/search?s=%22AbstractFormatter%22&pp=0&fl=Python&mp=1&ml=1&me=1&md=1&ff=1&filterChecked=true ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (merge 2.7 -> 2.7): Clean merge
On Tue, 13 Aug 2013 00:42:25 +0300 Ezio Melotti wrote: > > To avoid these big merges you can do: > # check the two heads that you are going to merge and their csids > hg heads . > # update to the other head (the one you pulled, not the one you committed) > hg up csid-of-the-other-head > # merge your changes on with the ones you pulled > hg merge > > This will merge the changes you just committed with the ones you > pulled, and result in a shorter diff that is easier to > read/review/merge. > Otherwise pulling and updating before committing will avoid the > problem entirely (unless you end up in a push-race). Or, if you are working on a single branch and no-one is watching you, you can do "hg pull --rebase". Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecating the formatter module
I never realized it existed till now. Considering the usually erratic projects I like do, I can see that coming in use in several in which I had to do odd workarounds. Keep it, but put better documentation. It's needed. Brett Cannon wrote: >At the PyCon CA sprint someone discovered the formatter module had >somewhat >low code coverage. We discovered this is because it's tested by >test_sundry, i.e. it's tested by importing it and that's it. > >We then realized that it isn't really used by anyone (pydoc uses it but >it >should have been using textwrap). Looking at the history of the module >it >has just been a magnet for cleanup revisions and not actual usage or >development since Guido added it back in 1995. > >I have created http://bugs.python.org/issue18716 to deprecate the >formatter >module for removal in Python 3.6 unless someone convinces me otherwise >that >deprecation and removal is the wrong move. > > > > >___ >Python-Dev mailing list >Python-Dev@python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >http://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks
Hi, Armin Rigo writes: > On Mon, Aug 12, 2013 at 9:39 AM, Arnaud Fontaine > wrote: >> Thread 1 is trying to import a module 'foo.bar' (where 'foo' is a >> package containing dynamic modules) handled by Import Hooks I >> implemented, so import lock is acquired before even running the hooks >> (Python/import.c:PyImport_ImportModuleLevel()). Then, these import >> hooks try to load objects from ZODB and a request is sent and handled >> by another thread (Thread 2) which itself tries to import another >> module. > > A quick hack might be to call imp.release_lock() and > imp.acquire_lock() explicitly, from your import hook code, around > calls to ZODB. I suggested the same in my initial email, but I was wondering if there could be any issue by releasing the lock in find_module()/load_module() until the module is actually added to sys.modules. Cheers, -- Arnaud Fontaine ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks
Brett Cannon writes: > On Mon, Aug 12, 2013 at 5:12 AM, Arnaud Fontaine wrote: >> Yes, I saw the bug report and its patch implementing the import lock per >> module (mentioned in my initial email) and watched the presentation by >> Brett Cannon (BTW, I could not find the diagram explained during the >> presentation, anyone knows if it's available somewhere?). > > http://prezi.com/mqptpza9xbic/?utm_campaign=share&utm_medium=copy Thanks. Is the full diagram only available somewhere? (I mean as an image or PDF file, not within the presentation document itself) Cheers, -- Arnaud Fontaine ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dealing with import lock deadlock in Import Hooks
On Tue, 13 Aug 2013 11:06:51 +0900 Arnaud Fontaine wrote: > Hi, > > Armin Rigo writes: > > On Mon, Aug 12, 2013 at 9:39 AM, Arnaud Fontaine > > wrote: > >> Thread 1 is trying to import a module 'foo.bar' (where 'foo' is a > >> package containing dynamic modules) handled by Import Hooks I > >> implemented, so import lock is acquired before even running the hooks > >> (Python/import.c:PyImport_ImportModuleLevel()). Then, these import > >> hooks try to load objects from ZODB and a request is sent and handled > >> by another thread (Thread 2) which itself tries to import another > >> module. > > > > A quick hack might be to call imp.release_lock() and > > imp.acquire_lock() explicitly, from your import hook code, around > > calls to ZODB. > > I suggested the same in my initial email, but I was wondering if there > could be any issue by releasing the lock in find_module()/load_module() > until the module is actually added to sys.modules. Well, you are obviously on your own with such hacks. There is a reason the lock exists. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
