Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-08 Thread Stefan Behnel
Eric Snow, 08.09.2013 00:22: On Mon, Sep 2, 2013 at 7:02 AM, Nick Coghlan wrote: The hook API I currently have in mind is a two step initialisation: PyImport_PrepareNAME (optional) PyImport_ExecNAME Should we also look at an API change for the initfunc() of PyImport_Inittab

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-07 Thread Eric Snow
On Mon, Sep 2, 2013 at 7:02 AM, Nick Coghlan ncogh...@gmail.com wrote: The hook API I currently have in mind is a two step initialisation: PyImport_PrepareNAME (optional) PyImport_ExecNAME Should we also look at an API change for the initfunc() of PyImport_Inittab entries?

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-06 Thread Stefan Behnel
Antoine Pitrou, 06.09.2013 07:43: Proxying wrapper? We shouldn't need that kind of tricks. The advantage is that it's controlled by the loader, and transparent to the module itself. Stefan ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-05 Thread Eric Snow
On Sat, Aug 24, 2013 at 7:07 AM, Stefan Behnel stefan...@behnel.de wrote: PEP 3121 would no longer be necessary. Extension types can do all we need. No more special casing of modules, that was the idea. One nice thing about PEP 3121 is the addition of md_state to module objects to store

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-05 Thread Antoine Pitrou
On Thu, 5 Sep 2013 23:26:31 -0600 Eric Snow ericsnowcurren...@gmail.com wrote: On Sat, Aug 24, 2013 at 7:07 AM, Stefan Behnel stefan...@behnel.de wrote: PEP 3121 would no longer be necessary. Extension types can do all we need. No more special casing of modules, that was the idea. One

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-05 Thread Eric Snow
On Sat, Aug 24, 2013 at 10:12 PM, PJ Eby p...@telecommunity.com wrote: I haven't had a chance to address this on the import-sig discussion yet about ModuleSpec, but I would like to just mention that one property of the existing module system that I'm not sure either this proposal or the

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-05 Thread Eric Snow
On Sat, Aug 31, 2013 at 1:16 PM, Stefan Behnel stefan...@behnel.de wrote: Nick Coghlan, 31.08.2013 18:49: This is actually my primary motivation for trying to improve the can this be reloaded or not? aspects of the loader API in PEP 451. I assume you mean that the extension module would be

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-05 Thread Eric Snow
On Mon, Sep 2, 2013 at 2:16 AM, Antoine Pitrou solip...@pitrou.net wrote: I think the biggest challenge here is to propose an API that's simple and easy to use (i.e. that doesn't make extension module writing more complicated than it currently is). +1 The basic concept of putting custom

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-02 Thread Antoine Pitrou
Le Sun, 1 Sep 2013 02:19:48 +1000, Nick Coghlan ncogh...@gmail.com a écrit : On 1 Sep 2013 00:10, Stefan Behnel stefan...@behnel.de wrote: *bump* Does this sound like a viable solution? This isn't likely to progress until we have Eric's PEP 451 to a point where it's ready for

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-02 Thread Nick Coghlan
On 2 September 2013 18:16, Antoine Pitrou solip...@pitrou.net wrote: Le Sun, 1 Sep 2013 02:19:48 +1000, Nick Coghlan ncogh...@gmail.com a écrit : On 1 Sep 2013 00:10, Stefan Behnel stefan...@behnel.de wrote: *bump* Does this sound like a viable solution? This isn't likely to progress

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-01 Thread Stefan Behnel
Nick Coghlan, 01.09.2013 03:28: On 1 Sep 2013 05:18, Stefan Behnel wrote: I can't really remember a case where I could afford the runtime overhead of implementing a wrapper in Python and going through something like ctypes or cffi. I mean, testing C libraries with Python tools would be one,

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-01 Thread Nick Coghlan
On 1 September 2013 18:11, Stefan Behnel stefan...@behnel.de wrote: Nick Coghlan, 01.09.2013 03:28: On 1 Sep 2013 05:18, Stefan Behnel wrote: I can't really remember a case where I could afford the runtime overhead of implementing a wrapper in Python and going through something like ctypes or

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-01 Thread Stefan Behnel
Nick Coghlan, 01.09.2013 14:23: That means the powers any new extension initialisation API will offer will be limited to: * letting the module know its own name (and other details) * letting the module explicitly block reloading * letting the module support loading multiple copies at once

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-01 Thread Antoine Pitrou
On Sun, 1 Sep 2013 11:28:36 +1000 Nick Coghlan ncogh...@gmail.com wrote: * PEP 3121 with a size of 0. As above, but avoids the module state APIs in order to support reloading. All module state (including type cross-references) is stored in hidden state (e.g. an instance of a custom type not

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-01 Thread Nick Coghlan
On 1 September 2013 23:03, Antoine Pitrou solip...@pitrou.net wrote: On Sun, 1 Sep 2013 11:28:36 +1000 Nick Coghlan ncogh...@gmail.com wrote: * PEP 3121 with a size of 0. As above, but avoids the module state APIs in order to support reloading. All module state (including type

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-01 Thread Antoine Pitrou
On Mon, 2 Sep 2013 00:10:08 +1000 Nick Coghlan ncogh...@gmail.com wrote: $ ./python Python 3.4.0a1+ (default:575071257c92+, Aug 25 2013, 00:42:17) [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] on linux Type help, copyright, credits or license for more information. import atexit

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Stefan Behnel
*bump* Does this sound like a viable solution? Stefan Stefan Behnel, 25.08.2013 14:36: Hi, thanks for bringing this up. It clearly shows that there is more to this problem than I initially thought. Let me just add one idea that your post gave me. PJ Eby, 25.08.2013 06:12: My

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Nick Coghlan
On 1 Sep 2013 00:10, Stefan Behnel stefan...@behnel.de wrote: *bump* Does this sound like a viable solution? This isn't likely to progress until we have Eric's PEP 451 to a point where it's ready for python-dev discussion and pronouncement. However, the revised loader API is being designed

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Nick Coghlan
Oops, had a draft from a few days ago that I was interrupted before sending. Finished editing the parts I believe are still relevant. On 25 Aug 2013 21:56, Stefan Behnel stefan...@behnel.de wrote: Nick Coghlan, 24.08.2013 23:43: On 25 Aug 2013 01:44, Stefan Behnel wrote: Nick Coghlan,

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Stefan Behnel
Nick Coghlan, 31.08.2013 18:49: On 25 Aug 2013 21:56, Stefan Behnel wrote: One key point to note is that it *doesn't* call _PyImport_FixupExtensionObject, which is the API that handles all the PEP 3121 per-module state stuff. Instead, the idea will be for modules that don't need additional C

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Antoine Pitrou
On Sat, 31 Aug 2013 21:16:10 +0200 Stefan Behnel stefan...@behnel.de wrote: Our experience is very different - my perspective is that the normal case either eschews C level global state in the extension module, because it causes so many problems, or else just completely ignores

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Stefan Behnel
Antoine Pitrou, 31.08.2013 21:27: On Sat, 31 Aug 2013 21:16:10 +0200 Stefan Behnel wrote: Our experience is very different - my perspective is that the normal case either eschews C level global state in the extension module, because it causes so many problems, or else just completely ignores

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Nick Coghlan
On 1 Sep 2013 05:18, Stefan Behnel stefan...@behnel.de wrote: Nick Coghlan, 31.08.2013 18:49: On 25 Aug 2013 21:56, Stefan Behnel wrote: One key point to note is that it *doesn't* call _PyImport_FixupExtensionObject, which is the API that handles all the PEP 3121 per-module state stuff.

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-25 Thread Nick Coghlan
On 25 August 2013 14:12, PJ Eby p...@telecommunity.com wrote: That is to say, in the pure PEP 302 world, there is no special status for reload that is different from load -- the *only* thing that's different is that there is already a module object to use, and there is *no guarantee that it's

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-25 Thread Stefan Behnel
Nick Coghlan, 24.08.2013 23:43: On 25 Aug 2013 01:44, Stefan Behnel wrote: Nick Coghlan, 24.08.2013 16:22: The new _PyImport_CreateAndExecExtensionModule function does the heavy lifting:

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-25 Thread Stefan Behnel
Hi, thanks for bringing this up. It clearly shows that there is more to this problem than I initially thought. Let me just add one idea that your post gave me. PJ Eby, 25.08.2013 06:12: My Importing package offers lazy imports by creating module objects in sys.modules that are a subtype of

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-25 Thread Terry Reedy
On 8/25/2013 7:54 AM, Stefan Behnel wrote: And what if you do from extmodule import some_function in a Python module? Then reloading couldn't replace that reference, just as for normal Python modules. Meaning that you'd still have to keep both modules properly alive in order to prevent crashes

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan
On 24 August 2013 15:51, Nick Coghlan ncogh...@gmail.com wrote: My current plan is to create an experimental prototype of this approach this weekend. That will include stdlib test cases, so it will also show how it looks from the extension developer's point of view. I prototyped as much as I

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Antoine Pitrou
On Sat, 24 Aug 2013 21:36:51 +1000 Nick Coghlan ncogh...@gmail.com wrote: On 24 August 2013 15:51, Nick Coghlan ncogh...@gmail.com wrote: My current plan is to create an experimental prototype of this approach this weekend. That will include stdlib test cases, so it will also show how it

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Stefan Behnel
Antoine Pitrou, 24.08.2013 13:53: This would also imply extension module have to be subclasses of the built-in module type. They can't be arbitrary objects like Stefan proposed. I'm not sure what the latter enables, but it would probably make things more difficult internally. My line of

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Antoine Pitrou
On Sat, 24 Aug 2013 14:51:42 +0200 Stefan Behnel stefan...@behnel.de wrote: Antoine Pitrou, 24.08.2013 13:53: This would also imply extension module have to be subclasses of the built-in module type. They can't be arbitrary objects like Stefan proposed. I'm not sure what the latter enables,

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Stefan Behnel
Antoine Pitrou, 24.08.2013 15:00: On Sat, 24 Aug 2013 14:51:42 +0200 Stefan Behnel wrote: Antoine Pitrou, 24.08.2013 13:53: This would also imply extension module have to be subclasses of the built-in module type. They can't be arbitrary objects like Stefan proposed. I'm not sure what the

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Stefan Behnel
Nick Coghlan, 24.08.2013 13:36: On 24 August 2013 15:51, Nick Coghlan wrote: My current plan is to create an experimental prototype of this approach this weekend. That will include stdlib test cases, so it will also show how it looks from the extension developer's point of view. I

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan
On 24 August 2013 23:19, Stefan Behnel stefan...@behnel.de wrote: Nick Coghlan, 24.08.2013 13:36: On 24 August 2013 15:51, Nick Coghlan wrote: My current plan is to create an experimental prototype of this approach this weekend. That will include stdlib test cases, so it will also show how it

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Stefan Behnel
Nick Coghlan, 24.08.2013 16:22: On 24 August 2013 23:19, Stefan Behnel wrote: Nick Coghlan, 24.08.2013 13:36: On 24 August 2013 15:51, Nick Coghlan wrote: My current plan is to create an experimental prototype of this approach this weekend. That will include stdlib test cases, so it will

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Terry Reedy
On 8/24/2013 8:51 AM, Stefan Behnel wrote: Antoine Pitrou, 24.08.2013 13:53: This would also imply extension module have to be subclasses of the built-in module type. They can't be arbitrary objects like Stefan proposed. I'm not sure what the latter enables, but it would probably make things

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Benjamin Peterson
2013/8/24 Terry Reedy tjre...@udel.edu: On 8/24/2013 8:51 AM, Stefan Behnel wrote: Antoine Pitrou, 24.08.2013 13:53: This would also imply extension module have to be subclasses of the built-in module type. They can't be arbitrary objects like Stefan proposed. I'm not sure what the latter

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan
On 25 Aug 2013 05:19, Benjamin Peterson benja...@python.org wrote: 2013/8/24 Terry Reedy tjre...@udel.edu: On 8/24/2013 8:51 AM, Stefan Behnel wrote: Antoine Pitrou, 24.08.2013 13:53: This would also imply extension module have to be subclasses of the built-in module type. They can't

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan
On 25 Aug 2013 01:44, Stefan Behnel stefan...@behnel.de wrote: Nick Coghlan, 24.08.2013 16:22: On 24 August 2013 23:19, Stefan Behnel wrote: Nick Coghlan, 24.08.2013 13:36: On 24 August 2013 15:51, Nick Coghlan wrote: My current plan is to create an experimental prototype of this

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread PJ Eby
On Fri, Aug 23, 2013 at 4:50 AM, Stefan Behnel stefan...@behnel.de wrote: Reloading and Sub-Interpreters == To reload an extension module, the module create function is executed again and returns a new module type. This type is then instantiated as by the original

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-23 Thread Antoine Pitrou
Hi, Le Fri, 23 Aug 2013 10:50:18 +0200, Stefan Behnel stefan...@behnel.de a écrit : Here's an initial attempt at a PEP for it. It is based on the (unfinished) ModuleSpec PEP, which is being discussed on the import-sig mailing list. Thanks for trying this. I think the PEP should contain

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-23 Thread Nick Coghlan
On 23 August 2013 19:18, Antoine Pitrou solip...@pitrou.net wrote: Hi, Le Fri, 23 Aug 2013 10:50:18 +0200, Stefan Behnel stefan...@behnel.de a écrit : Here's an initial attempt at a PEP for it. It is based on the (unfinished) ModuleSpec PEP, which is being discussed on the import-sig