On Sun, Aug 4, 2013 at 7:26 AM, Nick Coghlan <ncogh...@gmail.com> wrote: > On 4 August 2013 23:40, Eli Bendersky <eli...@gmail.com> wrote: >> On Sat, Aug 3, 2013 at 7:08 PM, Nick Coghlan <ncogh...@gmail.com> wrote: >>> Sure, it's just unusual to have a case where "importing is blocked by adding >>> None to sys.modules" differs from "not actually available", so I'd like to >>> understand the situation better. >> >> I must admit I'm confused by the behavior of import_fresh_module too. >> >> Snippet #1 raises the expected ImportError: >> >> sys.modules['pyexpat'] = None >> import _elementtree >> >> However, snippet #2 succeeds importing: >> >> ET = import_fresh_module('_elementtree', blocked=['pyexpat']) >> print(ET) > > /me goes and looks > > That function was much simpler when it was first created :P > > Still, I'm fairly confident the complexity of that dance isn't > relevant to the problem you're seeing. > >> I believe this happens because import_fresh_module does an import of >> the 'name' it's given before even looking at the blocked list. Then, >> it assigns None to sys.modules for the blocked names and re-imports >> the module. So in essence, this is somewhat equivalent to snippet #3: >> >> modname = '_elementtree' >> __import__(modname) >> del sys.modules[modname] >> for m in sys.modules: >> if modname == m or m.startswith(modname + '.'): >> del sys.modules[m] >> sys.modules['pyexpat'] = None >> ET = importlib.import_module(modname) >> print(ET) >> >> Which also succeeds. >> >> I fear I'm not familiar enough with the logic of importing to >> understand what's going on, but it has been my impression that this >> problem is occasionally encountered with import_fresh_module and C >> code that imports stuff (the import of pyexpat is done by C code in >> this case). > > I had missed it was a C module doing the import. Looking into the > _elementtree.c source, the problem in this case is the fact that a > shared library that doesn't use PEP 3121 style per-module state is > only loaded and initialised once, so reimporting it gets the same > module back (from the extension loading cache), even if the Python > level reference has been removed from sys.modules. Non PEP 3121 C > extension modules thus don't work properly with > test.support.import_fresh_module (as there's an extra level of caching > involved that *can't* be cleared from Python, because it would break > things). > > To fix this, _elementree would need to move the pyexpat C API pointer > to per-module state, rather than using a static variable (see > http://docs.python.org/3/c-api/module.html#initializing-c-modules). > With per-module state defined, the import machine should rerun the > init function when the fresh import happens, thus creating a new copy > of the module. However, this isn't an entirely trivial change for > _elementree, since: > > 1. Getting from the XMLParser instance back to the module where it was > defined in order to retrieve the capsule pointer via > PyModule_GetState() isn't entirely trivial in C. You'd likely do it > once in the init method, store the result in an XMLParser attribute, > and then tweak the EXPAT() using functions to include an appropriate > local variable definition at the start of the method implementation. > > 2. expat_set_error would need to be updated to accept the pyexpat > capsule pointer as a function parameter >
Thanks Nick; I suspected something of the sort is going on here, but you provide some interesting leads to look at. I'll probably open an issue to track this at some point. Eli _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com