[issue16392] import crashes on circular imports in ext modules

[Brett Cannon]

Change by Brett Cannon :


--
nosy:  -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Brett Cannon]

Brett Cannon added the comment:

https://www.python.org/dev/peps/pep-0489/ is trying to come up with a redesign 
of extension module loading and no one has submitted a patch for the 
documentation (although Stefan has inlined proposed wording in a comment).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[R. David Murray]

R. David Murray added the comment:

Yes, the resolution of this issue will be to add documentation.  Someone should 
turn Stefan's comment into a patch.

--
assignee:  - docs@python
components: +Documentation
nosy: +docs@python, r.david.murray
stage:  - needs patch
versions: +Python 3.5 -Python 3.2, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Omer Katz]

Omer Katz added the comment:

Is this issue resolved in any way?
Has there been a decision made on how to resolve it?
What's the status here?

--
nosy: +Omer.Katz

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Andrew Svetlov]

Changes by Andrew Svetlov andrew.svet...@gmail.com:


--
nosy: +asvetlov

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Eric Snow]

Changes by Eric Snow ericsnowcurren...@gmail.com:


--
nosy: +eric.snow

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Brett Cannon]

Brett Cannon added the comment:

The fully qualified name requirement is definitely a design flaw where init 
functions should just be given the module object with everything already set, 
just like what @importlib.util.module_for_loader does. Hopefully we can come up 
with a solution through your current discussion on python-dev.

As for the comment, yes at the top of the page makes sense. Whether example 
code should go somewhere else I don't know.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Stefan Behnel]

Stefan Behnel added the comment:

Agreed. Since it doesn't really fit into any specific function documentation, I 
would place it right at the top. Something like this:


The following functions can be used by C code to call into Python's import 
machinery.

Note that Python 3 does not automatically register an extension module in 
sys.modules on creation (see module.html#initializing-c-modules). It is only 
added after running through the whole module init function. This means that a 
request to import the current module while its init function is still running 
(either directly or transitively by other modules) will try to reimport the 
module. If you cannot be sure that this will not happen, you have to register 
the newly created module yourself as follows, using the fully qualified module 
name::

PyObject *the_module = PyModule_Create(py_module_def);
if (the_module == NULL) { /* failure ! */ };

PyObject *sys_modules = PyImport_GetModuleDict();
if (sys_modules == NULL) { /* failure ! */ };

if (PyDict_SetItemString(modules, the.package.and.module, the_module)  
0) { /* failure ! */ };


Maybe it should add another comment that this is a major quirk in the runtime 
and say sorry for being stupid here - I hope you can do better. Requiring the 
user to know the FQMN at build time because the import machinery fails to set 
it automatically is just embarrissing.

Then, after the first sentence in the module.html#initializing-c-modules 
section, I'd add this:


Note that, starting with Python 3.0, the module creation functions no longer 
register the module in sys.modules. The import machinery will only do this 
after the module init function has run. If you need to run imports as part of 
your module init function and happen to know the fully qualified module name in 
your code, it is best to register the module yourself after creating it.


I wonder if the code example shouldn't go on the module page.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Brett Cannon]

Brett Cannon added the comment:

I was thinking somewhere in http://docs.python.org/3/c-api/import.html since 
this only comes up when you try to execute an import during extension module 
initialization that involves a circular import.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Antoine Pitrou]

Antoine Pitrou added the comment:

 Maybe we should add a warning in some documentation somewhere about
 this and then close the issue?

I don't really know where to add it, in the C API docs perhaps,

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Stefan Behnel]

Stefan Behnel added the comment:

The problem is a) that the module does not necessarily know to which place it 
eventually gets installed (Cython relies on the distutils Extension not lying 
to it, for example, which people do from time to time), and b) that the call to 
Py_InitModule() only receives the plain module name, not the package path. And 
yes, as mentioned in the other issue, passing a pointer to a context 
description struct into the module init function would have been the right 
thing to change for Py3 and still is the right thing to change for Py4.

BTW, I can confirm that registering the module in sys.modules explicitly right 
after creation works around this issue. Given that Cython needs to know the 
FQMN at compile time anyway, this works for us. It still leaves the problem 
open for other extension code.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Brett Cannon]

Brett Cannon added the comment:

It sounds like Cython has its fix and CPython knows what should eventually be 
changed in Python 4 to bring extension module initialization closer to how 
Python module code is initialized.

Maybe we should add a warning in some documentation somewhere about this and 
then close the issue?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Stefan Behnel]

Stefan Behnel added the comment:

Since it's quite possible that this has nothing to do with the frozen part of 
the importlib specifically, I'm removing that bit from the ticket title.

--
title: frozen importlib crashes on circular imports in ext modules - import 
crashes on circular imports in ext modules

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Antoine Pitrou]

Antoine Pitrou added the comment:

Are you sure this worked before 3.3?

Regardless, it seems you could fix this issue by adding a single line just 
after the PyModule_Create() call:

  __pyx_m = PyModule_Create(__pyx_moduledef);
  #endif
  if (PyDict_SetItemString(PyImport_GetModuleDict(),
   __Pyx_NAMESTR(reimport), __pyx_m))
goto __pyx_L1_error;


The fundamental issue is that an extension is inserted into sys.modules 
only after its initialization function returns, so importing it recursively 
won't detect that it already exists.

(by contrast, a Python module is inserted into sys.modules before its code is 
executed inside the module's global namespace)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Antoine Pitrou]

Antoine Pitrou added the comment:

A simple test seems to confirm the problem already existed in 3.2.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Antoine Pitrou]

Antoine Pitrou added the comment:

Note that Py_InitModule4 in Python 2 did add the module object to sys.modules 
(by calling PyImport_AddModule) before returning, but PyModule_Create in Python 
3 doesn't.

(PEP 3121 doesn't seem to mention PyModule_Create at all, strangely; it seems 
the PEP doesn't follow the implementation)

--
nosy: +loewis
versions: +Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Stefan Behnel]

Stefan Behnel added the comment:

Hmm, we already do that for packages (i.e. when compiling __init__.py). Looks 
like this just needs to be done for all modules in Py3. And unless there is a 
compelling reason for Py_InitModule4() not to do it, I think it should be 
reverted to its Py2 behaviour.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Antoine Pitrou]

Antoine Pitrou added the comment:

I have detected a compatibility issue when reverting to the 2.x behaviour: 
extension modules which lie about their name in their PyModuleDef are broken. 
There are two such modules in 3.3: _io and _decimal.

Patch attached, anyway.

--
keywords: +patch
Added file: http://bugs.python.org/file27872/modulecreate.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Stefan Behnel]

Stefan Behnel added the comment:

Now that you mention it - wouldn't that suffer from not actually knowing the 
FQMN? There's that hack in the dynlib loader that stores the package context in 
a global variable so that the module creation function can figure it out. That 
might be a reason not to register the module automatically. Only the loader 
would know the correct package, not the module creation code.

Also see issue 13429.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Antoine Pitrou]

Antoine Pitrou added the comment:

 Now that you mention it - wouldn't that suffer from not actually
 knowing the FQMN? There's that hack in the dynlib loader that stores
 the package context in a global variable so that the module creation
 function can figure it out. That might be a reason not to register the 
 module automatically.

That's possible (although I would expect a module to know in which package it's 
supposed to live).
Another solution would have been to pass the module object to the init 
function, instead of letting the init function create it, but it's a bit too 
late (or too early :-)) to change the extension module init signature again.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com