Re: [Python-Dev] working with Python threads from C extension module?
On Fri, 2007-09-07 at 16:20 -0700, Bill Janssen wrote:
#define SSL_ALLOW_THREADS {if (_ssl_locks != NULL) { Py_BEGIN_ALLOW_THREADS
}}
#define SSL_DISALLOW_THREADS {if (_ssl_locks != NULL) {
Py_BEGIN_ALLOW_THREADS }}
I'd forgotten how convoluted Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS were. Anyone have any other suggestions about
how to do this?
Be convoluted yourself and do this:
#define PySSL_BEGIN_ALLOW_THREADS { if (_ssl_locks) { Py_BEGIN_ALLOW_THREADS
#define PySSL_END_ALLOW_THREADS Py_END_ALLOW_THREADS } }
(Untested, but I think it should work.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] working with Python threads from C extension module?
Be convoluted yourself and do this:
#define PySSL_BEGIN_ALLOW_THREADS { if (_ssl_locks) { Py_BEGIN_ALLOW_THREADS
#define PySSL_END_ALLOW_THREADS Py_END_ALLOW_THREADS } }
(Untested, but I think it should work.)
Yes, that had occurred to me. We want the code inside the braces
still to run if the locks aren't held, so something more like
#define PySSL_BEGIN_ALLOW_THREADS { \
PyThreadState *_save; \
if (_ssl_locks_count0) {_save = PyEval_SaveThread();}
#define PySSL_BLOCK_THREADS if
(_ssl_locks_count0){PyEval_RestoreThread(_save)};
#define PySSL_UNBLOCK_THREADS if (_ssl_locks_count0){_save =
PyEval_SaveThread()};
#define PySSL_END_ALLOW_THREADS if
(_ssl_locks_count0){PyEval_RestoreThread(_save);} \
}
would do the trick. Unfortunately, this doesn't deal with the macro
behaviour. The user has turned on threading; they expect reads and
writes to yield the GIL so that other threads can make progress. But
the fact that threading has been turned on after the SSL module has
been initialized, means that threads don't work inside the SSL code.
So the user's understanding of the system will be broken.
No, I don't see any good way to fix this except to add a callback
chain inside PyThread_init_thread, which is run down when threads are
initialized. Any module which needs to set up threads registers itself
on that chain, and gets called as part of PyThread_init_thread. But
I'm far from the smartest person on this list :-), so perhaps someone
else will see a good solution.
This has got to be a problem with other extension modules linked to
libraries which have their own threading abstractions.
Bill
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] working with Python threads from C extension module?
On 08/09/2007, Bill Janssen [EMAIL PROTECTED] wrote:
Be convoluted yourself and do this:
#define PySSL_BEGIN_ALLOW_THREADS { if (_ssl_locks) {
Py_BEGIN_ALLOW_THREADS
#define PySSL_END_ALLOW_THREADS Py_END_ALLOW_THREADS } }
(Untested, but I think it should work.)
Yes, that had occurred to me. We want the code inside the braces
still to run if the locks aren't held, so something more like
#define PySSL_BEGIN_ALLOW_THREADS { \
PyThreadState *_save; \
if (_ssl_locks_count0) {_save =
PyEval_SaveThread();}
#define PySSL_BLOCK_THREADS if
(_ssl_locks_count0){PyEval_RestoreThread(_save)};
#define PySSL_UNBLOCK_THREADS if (_ssl_locks_count0){_save =
PyEval_SaveThread()};
#define PySSL_END_ALLOW_THREADS if
(_ssl_locks_count0){PyEval_RestoreThread(_save);} \
}
would do the trick. Unfortunately, this doesn't deal with the macro
behaviour. The user has turned on threading; they expect reads and
writes to yield the GIL so that other threads can make progress. But
the fact that threading has been turned on after the SSL module has
been initialized, means that threads don't work inside the SSL code.
So the user's understanding of the system will be broken.
No, I don't see any good way to fix this except to add a callback
chain inside PyThread_init_thread, which is run down when threads are
initialized. Any module which needs to set up threads registers itself
on that chain, and gets called as part of PyThread_init_thread. But
I'm far from the smartest person on this list :-), so perhaps someone
else will see a good solution.
I think this is a helpful additional tool to solve threading problems.
Doesn't solve everything, but it certainly helps :-)
For instance, one thing it doesn't solve is when a library being wrapped can
be initialized with multithreading support, but only allows such
initialization as a very first API call; you can't initialize threading at
any arbitrary time during application runtime. Unfortunately I don't think
there is any sane way to fix this problem :-(
This has got to be a problem with other extension modules linked to
libraries which have their own threading abstractions.
Yes.
Another problem is that python extensions may not wish to incur performance
penalty of python threading calls. For instance, pyorbit has these macros:
#define pyorbit_gil_state_ensure() (PyEval_ThreadsInitialized()?
(PyGILState_Ensure()) : 0)
#define pyorbit_gil_state_release(state) G_STMT_START { \
if (PyEval_ThreadsInitialized())\
PyGILState_Release(state); \
} G_STMT_END
#define pyorbit_begin_allow_threads \
G_STMT_START { \
PyThreadState *_save = NULL;\
if (PyEval_ThreadsInitialized())\
_save = PyEval_SaveThread();
#define pyorbit_end_allow_threads \
if (PyEval_ThreadsInitialized())\
PyEval_RestoreThread(_save);\
} G_STMT_END
They all call PyEval_ThreadsInitialized() before doing anything thread
related to save some performance. The other reason to do it this way is
that the Python API calls themselves abort if they are called with threading
not initialized. It would be nice the upstream python GIL macros were more
like pyorbit and became no-ops when threading is not enabled.
--
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
The universe is always one step beyond logic. -- Frank Herbert
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] working with Python threads from C extension module?
This has got to be a problem with other extension modules linked to libraries which have their own threading abstractions. Sure enough, sqlite3 simply assumes threads (won't build without them), and turns them on if it's used (by calling PyThread_get_thread_ident(), which in turn calls PyThread_init_thread()). Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] working with Python threads from C extension module?
Reading through the C API documentation, I find:
``This is done so that dynamically loaded extensions compiled with
thread support enabled can be loaded by an interpreter that was
compiled with disabled thread support.''
I've currently got the set-up-SSL-threading code in _ssl.c surrounded
by a #ifdef HAVE_THREAD bracket. It sounds like that might not be
sufficient. It sounds like I need a runtime test for thread
availability, instead, like this:
#ifdef HAVE_THREAD
if (PyEval_ThreadsInitialized())
_setup_ssl_threads();
#endif
Seem right?
So what happens when someone loads the _ssl module, initializes the
threads, and tries to use SSL? It's going to start failing again. I
think I need my own version of Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS, don't I? Which also checks to see if the SSL
threading support has been initialized, in addition to the Python
threading support. Something like
#define SSL_ALLOW_THREADS {if (_ssl_locks != NULL) { Py_BEGIN_ALLOW_THREADS }}
#define SSL_DISALLOW_THREADS {if (_ssl_locks != NULL) { Py_BEGIN_ALLOW_THREADS
}}
Any comments?
Bill
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] working with Python threads from C extension module?
#define SSL_ALLOW_THREADS {if (_ssl_locks != NULL) { Py_BEGIN_ALLOW_THREADS }}
#define SSL_DISALLOW_THREADS {if (_ssl_locks != NULL) {
Py_BEGIN_ALLOW_THREADS }}
I'd forgotten how convoluted Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS were. Anyone have any other suggestions about
how to do this?
Raise an error if loaded in a non-threaded environment, then used in a
threaded environment? Dynamic initialization of threading?
Bill
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] working with Python threads from C extension module?
So what happens when someone loads the _ssl module, initializes the threads, and tries to use SSL? It's going to start failing again. I Which turns out to be exactly what test_ssl.py does. I'm tempted to have the _ssl module call PyEval_InitThreads(). Would that be kosher? Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] working with Python threads from C extension module?
Well, one shouldn't be bothering with threads unless the user intends to create threads. So I think it's not kosher. Once threads are initialized, everything runs a tad slower because the GIL manipulations actually cost time (even if there are no other threads). On 9/7/07, Bill Janssen [EMAIL PROTECTED] wrote: So what happens when someone loads the _ssl module, initializes the threads, and tries to use SSL? It's going to start failing again. I Which turns out to be exactly what test_ssl.py does. I'm tempted to have the _ssl module call PyEval_InitThreads(). Would that be kosher? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] working with Python threads from C extension module?
Well, one shouldn't be bothering with threads unless the user intends to create threads. So I think it's not kosher. Once threads are initialized, everything runs a tad slower because the GIL manipulations actually cost time (even if there are no other threads). I think that doing it in _ssl.c might be OK; it would only happen when the user loaded that extension module. In any case, I'm going to do it that way till we figure out a better solution. The alternatives right now are (1) let OpenSSL step all over itself (and potentially other things), or (2) remove the Py_BEGIN_ALLOW_THREADS on SSL context reads and writes. On 9/7/07, Bill Janssen [EMAIL PROTECTED] wrote: So what happens when someone loads the _ssl module, initializes the threads, and tries to use SSL? It's going to start failing again. I Which turns out to be exactly what test_ssl.py does. I'm tempted to have the _ssl module call PyEval_InitThreads(). Would that be kosher? The problem is the sequencing of the loading of the extension module, compared to when the user gets around to initializing threading. If we want to keep it kosher, we need a way to hook into PyEval_InitThreads() so that it will call the thread initialization routines of other dynamically loaded libraries that have already been loaded. Or a way to have Py_BEGIN_ALLOW_THREADS take into account that there may be more than one thread-dependent thing to check on. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
