[issue13429] provide __file__ to extension init function

[Stefan Behnel]

Stefan Behnel added the comment:

This has been resolved by PEP 489, issue 24268.
The module initialisation process receives the complete ModuleSpec now, 
starting with CPython 3.5, and can do with it whatever it likes, before 
executing any user code.

--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
versions: +Python 3.5 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Stefan Behnel]

Stefan Behnel added the comment:

Triggered discussion on python-dev:

http://thread.gmane.org/gmane.comp.python.devel/135764

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Stefan Behnel]

Stefan Behnel added the comment:

I'm increasing the target version as this didn't change anything for 3.3. 
However, for 3.4, it might be possible to work around this by splitting the 
module init function into two parts, one that gets executed in order to create 
the module object (and do safe C level setup etc.) and one that gets executed 
after properly registering the module and setting __file__ and friends 
externally and that can then populate the module dict etc.

I can see two ways how this can be done. The shared library could optionally 
export a second symbol (PyInit2_modulename or whatever), or it could set a 
sort of callback function on the module object it returns in its original init 
function, either in the module dict or using a C level interface. I prefer the 
latter because it's safer - code can't accidentally rely on this feature in 
other Python versions (or implementations) because it would fail to build there 
if the required C-API function isn't available.

A corner case that this wouldn't fix is the creation of more than one module 
object in the module init function, but I think it's a reasonable restriction 
that users should be on their own here and should properly set up these 
additional modules themselves. Alternatively, this could be handled (with some 
overhead) if the import machinery remembered all modules created during the 
execution of the module init function and post-processed them as noted above.

So, my proposal would be to add a new callback function field to module objects 
and a new C-API function PyModule_SetInitFunction(module, c_func) that 
registers a C callback function on a module object. Then, extend the shared 
library importing code to call that init function if it's available, passing it 
both the module instance and an extensible context struct in order to enable 
future extensions to this interface.

I realise that this might have to go through a PEP process, but do you see any 
obvious flaws I missed so far or reasons to reject this approach right away?

--
versions: +Python 3.4 -Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Stefan Behnel]

Stefan Behnel sco...@users.sourceforge.net added the comment:

Any comments on the last patch?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Stefan Behnel]

Stefan Behnel sco...@users.sourceforge.net added the comment:

As MvL noted in his response to issue 13431, simply adding a parameter to the 
module init function cannot safely be done before Python 4. So we are back to 
the idea of passing the information through to the module creation function, 
i.e. this very issue.

A variant of the implementation would be to store the context information in 
thread local storage instead of a global variable. That would work around any 
threading issues. However, this would not be required in the normal import 
case, only in the reinit case, as the import case is protected by the import 
lock, as we have seen. Personally, I do not consider this a good idea for the 
time being, since I doubt that the number of users for the reinitialisation API 
is currently worth caring about.

In any case, the semantics of __file__ for extension modules would basically 
become that __file__ refers to the last library that was loaded before calling 
the module init function. So all extension modules that this init function 
creates will inherit the same __file__. My guess is that they currently end up 
with no __file__ attribute at all, as the loader only sets it on the module 
that the init function returns. So I consider that an improvement already.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Stefan Behnel]

Stefan Behnel sco...@users.sourceforge.net added the comment:

I'm aware that these things happen, that's why I said it. Actually, wouldn't it 
rather be *correct* for __file__ to be set to the same file path for all 
modules that an extension module creates in its init function? That would 
suggest that _Py_ModuleImportContext shouldn't be set to NULL after the first 
assignment, but instead stay alive until it gets reset by the dynlib loader. If 
the loader gets invoked recursively later on, it will do the right thing by 
storing away the old value during the import and restoring it afterwards. So 
_Py_ModuleImportContext would always point to the path that contains the init 
function that is currently being executed.

Regarding the lock (which, I assume, is simply reentrant), it's being acquired 
far up when the import mechanism starts, so the dynlib loader and the init 
function call are protected.

Note that this does not apply to the reinit case. 
_PyImport_FindExtensionObject() does not acquire the lock itself (which seems 
correct), and it can be called directly from imp.init_builtin(), i.e. from user 
code. Maybe that's why the _Py_PackageContext protocol was not implemented 
there. That's rather unfortunate, though. I guess the reasoning is that new 
code that uses this new feature is expected to actually be reentrant, also in 
parallel, because the module it creates and works on is local to the current 
thread until the init function terminates. So the import lock is not strictly 
required here. This does complicate the __file__ feature, though, so the second 
(reinit) patch won't work as is. I think the right fix for Python 4 would be 
to simply pass a context struct into the module init function.

On a related note, I just stumbled over this code in 
_PyImport_FindExtensionObject():

else {
if (def-m_base.m_init == NULL)
return NULL;
mod = def-m_base.m_init();
if (mod == NULL)
return NULL;
PyDict_SetItem(PyImport_GetModuleDict(), name, mod);
Py_DECREF(mod);
}
if (_PyState_AddModule(mod, def)  0) {
PyDict_DelItem(PyImport_GetModuleDict(), name);
Py_DECREF(mod);
return NULL;
}

If PyDict_SetItem() fails, this is bound to crash. I think it would be worth 
looking into this mechanism a bit more.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Stefan Behnel]

Stefan Behnel sco...@users.sourceforge.net added the comment:

Updated patch that does not reset _Py_ModuleImportContext after use.

--
Added file: http://bugs.python.org/file23728/ext_module_init_file_path_2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Stefan Behnel]

Stefan Behnel sco...@users.sourceforge.net added the comment:

Replying to myself:
 I think the right fix for Python 4 would be to simply pass
 a context struct into the module init function.

Actually, this doesn't have to wait for Python 4. Changing the module init 
function to take a parameter should be backwards compatible in C. Existing code 
simply wouldn't read the value from the stack, and new (or updated) code could 
benefit from the feature, Cython code in particular.

Here is a follow-up ticket for this more general feature:

http://bugs.python.org/issue13431

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Jesús Cea Avión]

Changes by Jesús Cea Avión j...@jcea.es:


--
nosy: +jcea

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Stefan Behnel]

New submission from Stefan Behnel sco...@users.sourceforge.net:

In Python modules, the top-level module code sees the __file__ variable and can 
use it to refer to resources in package subdirectories, for example. This is 
not currently possible in extension modules, because __file__ is only set after 
running the module init function, and the module has no way to find out its 
runtime location.

CPython should set __file__ directly in PyModule_Create2(), based on 
information provided by the shared library loader. This would let 
PyModule_GetFilenameObject() work immediately with the newly created module 
object.

The relevant python-dev thread is here:

http://mail.python.org/pipermail/python-dev/2011-November/114476.html

A patch will follow soon.

--
components: Extension Modules, Interpreter Core
messages: 147881
nosy: scoder
priority: normal
severity: normal
status: open
title: provide __file__ to extension init function
type: feature request
versions: Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Stefan Behnel]

Changes by Stefan Behnel sco...@users.sourceforge.net:


--
keywords: +patch
nosy: +loewis
Added file: http://bugs.python.org/file23725/ext_module_init_file_path.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Stefan Behnel]

Stefan Behnel sco...@users.sourceforge.net added the comment:

Here is an extension to the patch that implements the protocol also for 
extension module reinitialisation, so that the module creation can also set 
__file__ and the proper package in that case. Currently without tests (and 
users, I guess).

--
Added file: http://bugs.python.org/file23726/ext_module_reinit_context.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc amaur...@gmail.com added the comment:

I suppose that the value of _Py_ModuleImportContext is protected by the import 
lock?

--
nosy: +amaury.forgeotdarc

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Stefan Behnel]

Stefan Behnel sco...@users.sourceforge.net added the comment:

I don't know how the import lock applies here. Would it have to be protected by 
it? The lifetime is restricted to the call of the extension module init 
function, and its value is saved recursively if the init function triggers 
further imports.

It works exactly like the existing _Py_PackageContext variable.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc amaur...@gmail.com added the comment:

But the GIL can be released in many places (e.g. a Py_DECREF), and another 
thread can enter the same function and update the same static variable.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Stefan Behnel]

Stefan Behnel sco...@users.sourceforge.net added the comment:

... and the module init function could create and register a different module 
first, and ...

Well, yes, it's a best effort thing. It's rather unlikely that the GIL would 
get released in between the call to the init function and the creation of the 
module within that function, but sure, I don't see a reason why it could not 
happen.

However, it can't happen in moduleobject.c between the NULL check and the 
setting of the __file__ attribute, so that is safe enough to not trigger 
crashes.

And even if the wrong __file__ value is set during the run of the init 
function, it will still get overwritten (and thus fixed) afterwards. So my 
intuition is that code that relies on this new feature will simply have to make 
sure the module object creation is the first thing it does, and code that does 
not rely on it, well, does not rely on it.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc amaur...@gmail.com added the comment:

 ... and the module init function could create and register a
 different module first, and ...
Actually, this *does* happen, the PIL module is written this way.
And I don't agree with the best effort argument.  If there is a slight chance 
that this does not work, we'll have to fix it sooner or later.

But please check this import lock thing: if _PyImport_AcquireLock and 
_PyImport_ReleaseLock are always called around extension module initialization, 
then we don't have to worry about other threads; and nested calls are already 
handled by the oldimportcontext variable in your patch.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Eric Snow]

Changes by Eric Snow ericsnowcurren...@gmail.com:


--
nosy: +eric.snow

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13429] provide __file__ to extension init function

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13429
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com