Re: [Python-Dev] multiple interpreters and extension modules
Jeremy Kloth schrieb: 1) is subclassing Python classes in C as a static type supported? Even if they would be declared on the heap, they would be bound to the first loaded Python class. As you found out: no, this isn't supported. To work around, you can wrap the extension module with Python module which invokes an explicit module initialization function; you then need to lookup the subclasses dynamically (going through thread_state-interp-modules). As to question 2, I'm in the process of prototyping that approach to see if it is feasible in our code and if so, (and people find it useful) I'll attempt to write a PEP for that feature. I'd like to see the module initialization/finalization be redone for Python 3, in particular with explicit finalization procedures. If Python 3 still supports multiple interpreters, this aspect should be taken into account. Whether there is any chance to add the PEP to 2.x, I don't know - it needs to be written first. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] multiple interpreters and extension modules
Jeremy Kloth schrieb: I think you understand exactly what is happening. It is happening for good reasons. Rather than asking for a change in semantics, I recommend that you deal with it, either in your Python code, or in your extension. It's not likely to change. I don't believe I was asking for a change in semantics, rather an additional, optional interface for extension module writers. And that *is* a change in semantics. The algorithm run by the interpreter on module startup will be different from what it is now, i.e. it is a behavior (semantic) change. I'll add here that it has been brought up here before that extension module finalization is a feature that would be appreciated. With that, it is not that far to add support for initialization/finalization for each interpreter. That is, of course, using a DllMain-like solution. I wouldn't like to see a DllMain-like solution. Instead, there should be a function vector, and a specification which function is called in what cases. There should be an entry point that typically just returns a pointer to this module vtable. With that approach in mind, I will be making changes so 4Suite will work in a production mod_python deployment (where the aforementioned error occurred). When that works, I'll come back with a proper PEP *and* patches against Python SVN to support its use. I hope no one was thinking I wanted someone else to do the work. I was actually going to, for several years now. Please do write the PEP before making the implementation. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] multiple interpreters and extension modules
[[ This may be somewhat c.l.p.-ish but I feel that this crossed into CPython development enough to merit posting here ]] I have received a bug report for 4Suite that involves a PyObject_IsInstance check failing for what appears to be the correct class, that is, the class names match. With some investigating, I have found that the true problem is with multiple interpreters. The reason for this is that each sub-interpreter has a new copy of any pure Python module. The following issues are also true for modules that have been reloaded, but I think that is common knowledge. I mention it only for completeness. Here is where it crosses into CPython development. Now extension modules do not get reloaded into each sub-interpreter, but a copy of the module's dict from when it was first loaded. This particular extension, when initialized, loads the class in question from the pure Python module and stores in in a global variable for quick access. I know I can change this specific case to reload the class each time and for this exact case, it is most likely what I'll do. However, we have other extensions that make use of this same technique and their performance would suffer greatly by doing the lookup for each use of the class. Also performance isn't the only issue here as we also have C types that inherit from a Python class and those types will obviously fail an isinstance() check in Python code run in different interpreters as the base class is frozen at the first import. So I guess the questions are: 1) is subclassing Python classes in C as a static type supported? Even if they would be declared on the heap, they would be bound to the first loaded Python class. 2) would it be worthwhile to have an extension loading function similar to DllMain() for Windows' DLLs which is called multiple times with a flag stating the reason for the call? For reference, see: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/dllmain.asp I envision the reasons as: 'module init', 'module fini', 'interpreter init', 'interpreter fini'. I see no current need for special treatment of threads as they share the same interpreter space. As to question 2, I'm in the process of prototyping that approach to see if it is feasible in our code and if so, (and people find it useful) I'll attempt to write a PEP for that feature. My current thoughts on that are instead of the current initmodule name entry point in C modules, a new entry point is defined, and if found, used in its place. That way, this proposal would be optional for extension writers and existing modules would continue to work without change. Feedback appreciated. -- Jeremy Kloth http://4suite.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] multiple interpreters and extension modules
Jeremy Kloth [EMAIL PROTECTED] wrote: [[ This may be somewhat c.l.p.-ish but I feel that this crossed into CPython development enough to merit posting here ]] I have received a bug report for 4Suite that involves a PyObject_IsInstance check failing for what appears to be the correct class, that is, the class names match. With some investigating, I have found that the true problem is with multiple interpreters. The reason for this is that each sub-interpreter has a new copy of any pure Python module. The following issues are also true for modules that have been reloaded, but I think that is common knowledge. I mention it only for completeness. If I remember correctly, Python allows you to use multiple interpreters in the same process, but it makes no guarantees as to their correctness when running. See this post for further discussion on the issue: http://mail.python.org/pipermail/python-list/2004-January/244343.html You can also search for 'multiple python interpreters single process' in google without quotes to hear people lament over the (generally broken) multiple Python interpreter support. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] multiple interpreters and extension modules
On Friday 22 December 2006 5:02 pm, Josiah Carlson wrote: Jeremy Kloth [EMAIL PROTECTED] wrote: [[ This may be somewhat c.l.p.-ish but I feel that this crossed into CPython development enough to merit posting here ]] I have received a bug report for 4Suite that involves a PyObject_IsInstance check failing for what appears to be the correct class, that is, the class names match. With some investigating, I have found that the true problem is with multiple interpreters. The reason for this is that each sub-interpreter has a new copy of any pure Python module. The following issues are also true for modules that have been reloaded, but I think that is common knowledge. I mention it only for completeness. If I remember correctly, Python allows you to use multiple interpreters in the same process, but it makes no guarantees as to their correctness when running. See this post for further discussion on the issue: http://mail.python.org/pipermail/python-list/2004-January/244343.html You can also search for 'multiple python interpreters single process' in google without quotes to hear people lament over the (generally broken) multiple Python interpreter support. The problem here is that it is mod_python using the multiple interpreters. We have no control over that. What I'm proposing is fixing the extension module support for multiple interpreters with the bonus of adding extension module finalization which I've seen brought up here before. Fixing this does require support by the extension module author, but if that author doesn't feel the need to work in mod_python (if, of course, they load module level constants), that is their loss. Is 4Suite that different in its use of hybrid Python and C extensions? There is lots of back and forth between the two layers and performance is critical. I really don't feel like recoding thousands of lines of Python code into C just to get 4Suite to work in mod_python without error. -- Jeremy Kloth http://4suite.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] multiple interpreters and extension modules
I think you understand exactly what is happening. It is happening for good reasons. Rather than asking for a change in semantics, I recommend that you deal with it, either in your Python code, or in your extension. It's not likely to change. --Guido On 12/22/06, Jeremy Kloth [EMAIL PROTECTED] wrote: [[ This may be somewhat c.l.p.-ish but I feel that this crossed into CPython development enough to merit posting here ]] I have received a bug report for 4Suite that involves a PyObject_IsInstance check failing for what appears to be the correct class, that is, the class names match. With some investigating, I have found that the true problem is with multiple interpreters. The reason for this is that each sub-interpreter has a new copy of any pure Python module. The following issues are also true for modules that have been reloaded, but I think that is common knowledge. I mention it only for completeness. Here is where it crosses into CPython development. Now extension modules do not get reloaded into each sub-interpreter, but a copy of the module's dict from when it was first loaded. This particular extension, when initialized, loads the class in question from the pure Python module and stores in in a global variable for quick access. I know I can change this specific case to reload the class each time and for this exact case, it is most likely what I'll do. However, we have other extensions that make use of this same technique and their performance would suffer greatly by doing the lookup for each use of the class. Also performance isn't the only issue here as we also have C types that inherit from a Python class and those types will obviously fail an isinstance() check in Python code run in different interpreters as the base class is frozen at the first import. So I guess the questions are: 1) is subclassing Python classes in C as a static type supported? Even if they would be declared on the heap, they would be bound to the first loaded Python class. 2) would it be worthwhile to have an extension loading function similar to DllMain() for Windows' DLLs which is called multiple times with a flag stating the reason for the call? For reference, see: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/dllmain.asp I envision the reasons as: 'module init', 'module fini', 'interpreter init', 'interpreter fini'. I see no current need for special treatment of threads as they share the same interpreter space. As to question 2, I'm in the process of prototyping that approach to see if it is feasible in our code and if so, (and people find it useful) I'll attempt to write a PEP for that feature. My current thoughts on that are instead of the current initmodule name entry point in C modules, a new entry point is defined, and if found, used in its place. That way, this proposal would be optional for extension writers and existing modules would continue to work without change. Feedback appreciated. -- Jeremy Kloth http://4suite.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] multiple interpreters and extension modules
On 12/23/06, Jeremy Kloth [EMAIL PROTECTED] wrote: On Friday 22 December 2006 5:02 pm, Josiah Carlson wrote: Jeremy Kloth [EMAIL PROTECTED] wrote: [[ This may be somewhat c.l.p.-ish but I feel that this crossed into CPython development enough to merit posting here ]] I have received a bug report for 4Suite that involves a PyObject_IsInstance check failing for what appears to be the correct class, that is, the class names match. With some investigating, I have found that the true problem is with multiple interpreters. The reason for this is that each sub-interpreter has a new copy of any pure Python module. The following issues are also true for modules that have been reloaded, but I think that is common knowledge. I mention it only for completeness. If I remember correctly, Python allows you to use multiple interpreters in the same process, but it makes no guarantees as to their correctness when running. See this post for further discussion on the issue: http://mail.python.org/pipermail/python-list/2004-January/244343.html You can also search for 'multiple python interpreters single process' in google without quotes to hear people lament over the (generally broken) multiple Python interpreter support. The problem here is that it is mod_python using the multiple interpreters. We have no control over that. What I'm proposing is fixing the extension module support for multiple interpreters with the bonus of adding extension module finalization which I've seen brought up here before. Fixing this does require support by the extension module author, but if that author doesn't feel the need to work in mod_python (if, of course, they load module level constants), that is their loss. Is 4Suite that different in its use of hybrid Python and C extensions? There is lots of back and forth between the two layers and performance is critical. I really don't feel like recoding thousands of lines of Python code into C just to get 4Suite to work in mod_python without error. It's a whole lot more practical to just stop using mod_python and go for one of the other ways of exposing Python code to the internet. I bet you can get the same or better performance out of another solution anyway, and you'd save deployment headaches. -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] multiple interpreters and extension modules
On Friday 22 December 2006 7:54 pm, Bob Ippolito wrote: It's a whole lot more practical to just stop using mod_python and go for one of the other ways of exposing Python code to the internet. I bet you can get the same or better performance out of another solution anyway, and you'd save deployment headaches. I have no control over end-users choice of Python/webserver integration, I just end up making it possible to run our software in the environment of *their* choice. If it is the opinion that it is mod_python that is broken, I'd gladly point the users to the location stating that fact/belief. It would make my life easier. -- Jeremy Kloth http://4suite.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] multiple interpreters and extension modules
On Friday 22 December 2006 7:16 pm, Guido van Rossum wrote: I think you understand exactly what is happening. It is happening for good reasons. Rather than asking for a change in semantics, I recommend that you deal with it, either in your Python code, or in your extension. It's not likely to change. I don't believe I was asking for a change in semantics, rather an additional, optional interface for extension module writers. 2) would it be worthwhile to have an extension loading function similar to DllMain() for Windows' DLLs which is called multiple times with a flag stating the reason for the call? For reference, see: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/ base/dllmain.asp I envision the reasons as: 'module init', 'module fini', 'interpreter init', 'interpreter fini'. I see no current need for special treatment of threads as they share the same interpreter space. As to question 2, I'm in the process of prototyping that approach to see if it is feasible in our code and if so, (and people find it useful) I'll attempt to write a PEP for that feature. I'll add here that it has been brought up here before that extension module finalization is a feature that would be appreciated. With that, it is not that far to add support for initialization/finalization for each interpreter. That is, of course, using a DllMain-like solution. My current thoughts on that are instead of the current initmodule name entry point in C modules, a new entry point is defined, and if found, used in its place. That way, this proposal would be optional for extension writers and existing modules would continue to work without change. With that approach in mind, I will be making changes so 4Suite will work in a production mod_python deployment (where the aforementioned error occurred). When that works, I'll come back with a proper PEP *and* patches against Python SVN to support its use. I hope no one was thinking I wanted someone else to do the work. -- Jeremy Kloth http://4suite.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] multiple interpreters and extension modules
On 12/23/06, Jeremy Kloth [EMAIL PROTECTED] wrote: On Friday 22 December 2006 7:54 pm, Bob Ippolito wrote: It's a whole lot more practical to just stop using mod_python and go for one of the other ways of exposing Python code to the internet. I bet you can get the same or better performance out of another solution anyway, and you'd save deployment headaches. I have no control over end-users choice of Python/webserver integration, I just end up making it possible to run our software in the environment of *their* choice. If it is the opinion that it is mod_python that is broken, I'd gladly point the users to the location stating that fact/belief. It would make my life easier. Well, it clearly is broken wrt pure python modules and objects that persist across requests. I believe that it's also broken with any extension that uses the PyGILState API due to the way it interacts with multiple interpreters. I stopped using mod_python years ago due to the sorts of issues that you're bringing up here (plus problems compiling, deploying, RAM bloat, etc.). I don't have any recent experience or references that I can point you to, but I can definitely say that I have had many good experiences with the WSGI based solutions (and Twisted, but that's a different game). I would at least advise your user that there are several perfectly good ways to make Python speak HTTP, and mod_python is the only one with this issue. -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
