Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-28 Thread Nick Coghlan
On Mon, 28 Jan 2019 at 19:36, Stephan Reiter wrote: > > Reading through that post, I think I have everything covered but this here: > - The third and final scenario, and the one where the extended GIL > state functions for Ensure is still required, is where code doesn't > have the GIL as yet and

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-28 Thread Stephan Reiter
Reading through that post, I think I have everything covered but this here: - The third and final scenario, and the one where the extended GIL state functions for Ensure is still required, is where code doesn't have the GIL as yet and wants to make a call into sub interpreter rather than the main

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-28 Thread Nick Coghlan
On Mon, 28 Jan 2019 at 00:32, Stephan Reiter wrote: > > Cool. Thanks, Nick! > > I did experiments based on this idea > (https://github.com/stephanreiter/cpython/commit/3bca91c26ac81e517b4aa22302be1741b3315622) > and haven't rejected it yet. :-) After talking to Graham about this, I

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-27 Thread Gregory P. Smith
On Thu, Jan 24, 2019 at 1:25 PM Chris Barker - NOAA Federal via Python-Dev < python-dev@python.org> wrote: > If your primary concern is module clashes between plugins, maybe you > can hack around that: > > 1) if the plugins are providing copies of any other modules, then you > can simply require

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-27 Thread Nathaniel Smith
On Sun, Jan 27, 2019, 06:34 Stephan Reiter Cool. Thanks, Nick! > > I did experiments based on this idea ( > https://github.com/stephanreiter/cpython/commit/3bca91c26ac81e517b4aa22302be1741b3315622) > and haven't rejected it yet. :-) > > Together with the other fix ( >

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-27 Thread Stephan Reiter
Cool. Thanks, Nick! I did experiments based on this idea ( https://github.com/stephanreiter/cpython/commit/3bca91c26ac81e517b4aa22302be1741b3315622) and haven't rejected it yet. :-) Together with the other fix (

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-27 Thread Nick Coghlan
On Thu, 24 Jan 2019 at 05:45, Stephan Reiter wrote: > If we create a fresh OS thread and make it call PyGILState_Ensure, it > won't have a PyThreadState saved under autoTLSkey. That means it will > create one using the main interpreter. I, as the developer embedding > Python into my application

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-24 Thread Chris Barker - NOAA Federal via Python-Dev
If your primary concern is module clashes between plugins, maybe you can hack around that: 1) if the plugins are providing copies of any other modules, then you can simply require them to put them in their own namespace — that is, a plug-in is a single package, with however many sub modules as it

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-23 Thread Stephan Reiter
You all do make me feel very welcome in this community! Thank you very much! :-) And thank you for all the thought and time you put into your message, Eric. I do appreciate in particular all the alternatives you presented; you provide a good picture of my options. Not ruling out any of them, I'll

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-23 Thread Eric Snow
Hi Stephan, On Tue, Jan 22, 2019 at 9:25 AM Stephan Reiter wrote: > I am new to the list and arriving with a concrete problem that I'd > like to fix myself. That is great! Statements like that are a good way to get folks interested in your success. :) > I am embedding Python (3.6) into my C++

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-23 Thread Stephan Reiter
Hi! Well, the plugins would be created by third-parties and I'd like them to enable bunding of modules with their plugins. I am afraid of modules with the same name, but being different, or different versions of modules being used by different plugins. If plugins share an interpreter, the module

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-23 Thread Petr Viktorin
On 1/23/19 3:33 AM, Stephan Reiter wrote: Thanks for the answers so far. I appreciate them! Nathaniel, I'd like to allow Python plugins in my application. A plugin should be allowed to bring its own modules along (i.e. plugin-specific subdir is in sys.path when the plugin is active) and hence

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-22 Thread Nathaniel Smith
On Tue, Jan 22, 2019 at 6:33 PM Stephan Reiter wrote: > > Thanks for the answers so far. I appreciate them! > > Nathaniel, I'd like to allow Python plugins in my application. A > plugin should be allowed to bring its own modules along (i.e. > plugin-specific subdir is in sys.path when the plugin

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-22 Thread Stephan Reiter
Thanks for the answers so far. I appreciate them! Nathaniel, I'd like to allow Python plugins in my application. A plugin should be allowed to bring its own modules along (i.e. plugin-specific subdir is in sys.path when the plugin is active) and hence some isolation of them will be needed, so

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-22 Thread Nathaniel Smith
There are currently numerous incompatibilities between numpy and subinterpreters, and no concrete plan for fixing them. The numpy team does not consider subinterpreters to be a supported configuration, and can't help you with any issues you run into. I know the concept of subinterpreters is really

Re: [Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-22 Thread Antoine Pitrou
On Tue, 22 Jan 2019 15:32:22 +0100 Stephan Reiter wrote: > > Numpy's extension umath calls PyGILState_Ensure(), which in turn calls > PyEval_RestoreThread on the (auto) threadstate of the main > interpreter. And that's wrong. > We are already holding the GIL with the threadstate of our current >

[Python-Dev] Sub-interpreters: importing numpy causes hang

2019-01-22 Thread Stephan Reiter
Hi all! I am new to the list and arriving with a concrete problem that I'd like to fix myself. I am embedding Python (3.6) into my C++ application and I would like to run Python scripts isolated from each other using sub-interpreters. I am not using threads; everything is supposed to run in the