[jira] Commented: (MODPYTHON-217) Python 2.3 and simplified GIL state API still causes problems.
[
https://issues.apache.org/jira/browse/MODPYTHON-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12523668
]
Deron Meranda commented on MODPYTHON-217:
-
Another real-world case where this comes up is with the Xapian
search engine (http://xapian.org/). It also is SWIG generated.
See their bug #185,
http://www.xapian.org/cgi-bin/bugzilla/show_bug.cgi?id=185
Python 2.3 and simplified GIL state API still causes problems.
--
Key: MODPYTHON-217
URL: https://issues.apache.org/jira/browse/MODPYTHON-217
Project: mod_python
Issue Type: Bug
Components: core
Affects Versions: 3.3.x, 3.2.10
Reporter: Graham Dumpleton
There are still problems in mod_python when third party extension modules are
used which make use of the Python simplified GIL state API. Specifically, if
C code makes calls to PyGILState_Ensure() when it already holds the Python
GIL and has an active thread state, but the active thread was one which was
originally created outside of Python, such as by Apache with mod_python, then
the call to PyGILState_Ensure() results in a dead lock. This is occurring
because the PyGILState_Ensure() function finds it does not know anything
about the current thread ID and therefore assumes it needs to create a new
thread state for it and make that the active thread state. In doing that
though it tries to acquire the GIL when the thread already holds it,
resulting in the deadlock.
At this stage it is believed this only occurs with Python 2.3 and shouldn't
be a problem with later versions of Python. This is the case as later
versions of Python were modified so that PyThreadState_New() will register
the thread with the underpinnings of the PyGILState_???() mechanism and
because it already knows about it, it will not attempt to create a new thread
state object and so the deadlock will not occur.
When C code is hand crafted it is unlikely that anyone would call
PyGILState_Ensure() when they already know they hold the GIL, ie., when
calling out of Python code, but SWIG when used with the -threads option does
exactly this. For example:
SWIGINTERN PyObject *_wrap_ap_get_server_version(PyObject
*SWIGUNUSEDPARM(self), PyObject *args) { PyObject *resultobj = 0;
char *result = 0 ;
SWIG_PYTHON_THREAD_BEGIN_BLOCK;
if (!PyArg_ParseTuple(args,(char *):ap_get_server_version)) SWIG_fail;
{
SWIG_PYTHON_THREAD_BEGIN_ALLOW;
result = (char *)ap_get_server_version();
SWIG_PYTHON_THREAD_END_ALLOW;
}
resultobj = SWIG_FromCharPtr((const char *)result);
SWIG_PYTHON_THREAD_END_BLOCK;
return resultobj;
fail:
SWIG_PYTHON_THREAD_END_BLOCK;
return NULL;
}
where SWIG_PYTHON_THREAD_BEGIN_BLOCK eventually expands to a call to
PyGILState_Ensure().
The only solution to this problem would be for mod_python to behave
differently when it is acquiring a thread state against the main Python
interpreter, ie. 'main_interpreter'. Specifically, rather than use
PyThreadState_New() etc, it should call PyGILState_Ensure() instead. When
needing to release the main Python interpreter it should call
PyGILState_Release(). It should continue to work as before for all other
interpreters and anyone with GILState code would need to ensure they are
using 'main_interpreter'.
Unfortunately at the moment making this change is not completely straight
forward as when an interpreter is being released it isn't known that it is
the main Python interpreter. Thus some knowledge that it is the main
interpreter would need to be carried into the call. The only other option is
that the list of all active interpreters is traversed and the last one in the
list is assumed to be the main interpreter. Then if the interpreter being
released matches that then act differently.
Note that at present the acquire/release methods are exported as optional
functions so that other Apache modules use them. It is unlikely that anyone
is making use of them, so changing the in/out parameters of the functions
would possibly be acceptable.
Also note that maintaining support for threaded and non threaded Python and
now this GIL state variation is going to make the code even more messy. As
such it might be worthwhile considering dropping support for non threaded
Python. These days Python defaults to being threaded anyway, and highly
unlikely that someone would want to use non thread Python with mod_python
anyway. When ever any development of mod_python is done, no testing is ever
done with a non threaded Python anyway and so it is possibly questionable if
that support still works.
FWIW, this issue came up as a result of queries expressed in:
http://www.modpython.org/pipermail/mod_python/2007-April/023445.html
The GIL state issue has
Re: [Mod_python Wiki] Update of LocalSpellingWords by MartinStoufer
On 12/4/06, Jim Gallacher [EMAIL PROTECTED] wrote: Can someone explain the significance of this? The following page has been changed by MartinStoufer: http://wiki.apache.org/mod_python/LocalSpellingWords ... + programattic The wiki has a special page which contains exceptions for a spell-check dictionary. Whenever anybody is editing a wiki page they can choose to have the server spell check the page before saving it. Normally the wiki knows about most English words, but you can also add words to the LocalSpellingWords page to add more. However the LocalSpellingWords is global across the whole wiki regardless of who is editing. It looks like Martin was spell checking and accidentally added the word programattic to the exception list rather than correcting the misspelled word. It does no harm, except programattic will no longer be flagged as a mis-spelled word. Normally LocalSpellingWords is useful if there are some common invented words that are used frequently in the wiki. mod_python would be a perfect example, but the underscore in it already causes it to be ignored during spell checking. I can clean up the mistaken entry later, unless somebody beats me to it (it's just a wiki page, so it can be edited). -- Deron Meranda
Re: For Consideration: MySQLSession support
On 8/9/06, Nicolas Lehuen [EMAIL PROTECTED] wrote:
I think MySQL has also the support for INSERT OR UPDATE requests which are
quite handy when implementing do_save.
You probably mean the REPLACE statement,
http://www.mysql.org/doc/refman/5.1/en/replace.html
If you want something a little more like standard SQL like, you can
also use something a little more pythonic which eliminates the need
to do a select statement first,
try:
c.execute( insert into )
except MySQLdb.IntegrityError, err:
if e.args[0] == 1062: # duplicate key
c.execute( update . )
else:
raise
Unfortunately the MySQLdb module doesn't seem to provide
any identifiers for the magic constant numbers like 1062.
Also you may want to (should) use transactions, especially for
the methods which execute multiple SQL statements. This will
help eliminate some of the race conditions that exist in your
code. Something like the following helper methods can be useful:
def __init_(self, ...):
self.in_trans = False
def start_transaction(self):
Begin a database transaction.
You must later call either commit_transaction() or
rollback_transaction().
import datetime
if self.in_trans:
raise RuntimeError('Attempted to start a nested transaction')
self.db.autocommit(False)
c = self.db.cursor() # Yes, use a private local cursor here
c.execute(start transaction)
self.in_trans = True
c.close()
def commit_transaction(self):
Commit the current database transaction.
The transaction should have started with the start_transaction() method.
if not self.in_trans:
raise RuntimeError('Attempted to commit before a
transaction was started')
db.commit()
self.in_trans = False
db.autocommit(True)
def rollback_transaction(self):
Roll back the current database transaction.
The transaction should have started with the start_transaction() method.
if not self.in_trans:
raise RuntimeError('Attempted to rollback before a
transaction was started')
db.rollback()
self.in_trans = False
db.autocommit(True)
And then in your methods that use multple SQL statements use
a code pattern like:
def some_complex_method(self):
success = False
self.start_transaction()
try:
try:
cursor.execute( statement 1 )
cursor.execute( statement 2 )
cursor.execute( statement 3 )
except:
success = False
raise
else:
success = True
finally:
if success:
self.commit_transaction()
else:
self.rollback_transaction()
return
Supposedly in a future version of python you'll be able to
collapse the nested (try: try: except: finally:) into a single
try block.
--
Deron Meranda
Re: Documentation for PythonOption
On 4/22/06, Graham Dumpleton [EMAIL PROTECTED] wrote: Unfortunately it tends to be just you and I tossing ideas back and forth and when you have a two person democracy at work, hard to get some consensus of what is the best direction to go. :-( I think the namespace direction you're taking is great. Just to be clear, the dotted-notation is simply that, a notation. The interface to req.get_options() is not changing is it? I also think that a listing of all the reserved or mod_python options in the docs for PythonOption is a very good idea. And it should definitely be mentioned that the entire mod_python.* names are reserved, and that applications or frameworks are encouraged to implement their own prefix/namespaces for their own options. -- Deron Meranda
Re: mysqldb error
On 4/12/06, Firat KUCUK [EMAIL PROTECTED] wrote:
conn = MySQLdb.connect(
host = '127.0.0.1',
user = 'pismikrop',
passwd = 'pass',
db = 'db')
Unrelated to your question, but I see this a lot. You should
not hardcode the database password in your script. If your
script ever fails you may end up with a Python traceback, and
if PythonDebug is enabled (or you're using a framework on
top of mod_python) parts of your source code could be
dumped across the HTML output, including your password!
In web scripts, you should always read any passwords, etc.,
from some other place such as a file.
pw = open('.htdbpass','r').readline().strip()
conn = MySQLdb.connect(
host = '127.0.0.1',
user = 'pismikrop',
passwd = pw,
db = 'db')
--
Deron Meranda
Re: mysqldb error
OperationalError: (2003, Can't connect to MySQL server on '10.0.0.6' (111)) Error 111 is a socket connection refused (ECONNREFUSED). So most likely this has nothing to do with your db username or password. But the ip address does look unusual. First, does the ip 10.0.0.6 mean anything? Is it an address of one of your interfaces? Use /sbin/ip addr list to find out. Do you have SELinux enabled? Run /usr/sbin/sestatus Do you have any iptables firewall rules that may be blocking port 3306? -- Deron Meranda
Re: Pickling/unpickling top-level functions, classes etc.
On 3/29/06, Graham Dumpleton [EMAIL PROTECTED] wrote: Are you okay with: http://issues.apache.org/jira/browse/MODPYTHON-81 Pickling/unpickling top-level functions defined in published module no longer works in mod_python 3.2 being resolved as Won't Fix and then closed? I agree that this seems to be something that is just not solvable without causing complete havoc with all the specialized import and reload functionality, or resulting in a solution that is too fragile. It is just a limitation of the pickle mechanism. This of course doesn't mean that users wouldn't want to pickle these kinds of things. But that the burden in those cases should be on them. It may be possible to solve this for class instances (e.g., objects) by subclassing the Unpickler class and substituting a smarter find_class() method. But as for globals, functions, etc., it looks like it may be much harder. The user may also be able to take advantage of the external object pickling (with persistent ids), but I haven't looked at them too closely. Regardless, there are lots of alternatives, so I have no problem with mod_python not solving this one (although the mod_python documentation should clearly emphasize these pickling limitiations). -- Deron Meranda
Re: Vote on whether to integrate server side include (SSI) support.
+1 (although I'm indifferent about into which release it goes) -- Deron Meranda
Re: [DRAFT] ANNOUNCE: Mod_python 3.2.7
Will you also be putting the new 3.2.7 HTML/PDF docs on http://modpython.org/ ? It also might be useful if you provide a link to Appendix C along with the announcement so people can easily find the what's new information. Oh, and for the older docs on the website, is the 3.2 documentation really release 3.2? Because the first HTML page says 3.1.3. And while you're updating the website, what about http://issues.apache.org/jira/browse/MODPYTHON-53 The link to SVN is now there, but the JIRA link is not. Being able to at least browse the issue list can be important for users, even if they are not on the python-dev list. -- Deron Meranda
Re: bash bug (was Re: mod_python 3.2.7 available for testing)
On 2/6/06, Jim Gallacher [EMAIL PROTECTED] wrote: A couple of thoughts on this issue. According to the gentoo bug report quoted below, the problem in configure.in is the double backslash escape sequence in the line: MP_VERSION=`echo $MP_VERSION | sed s/\\//g` Changing this to: MP_VERSION=`echo $MP_VERSION | sed s/\//g` fixes it for bash 3.1. I wonder why we are using \\ since the gentoo fix seems to work ok with bash 3.0 (and GNU sed) on my system just as well. Is it there to support other shells, other sed versions, older bash versions... ?? I suggest we either adopt the gentoo fix, or avoid the problem altogether by using tr. eg. MP_VERSION=`echo $MP_VERSION | tr -d ''` I'm assuming tr is always available on UNIX-like systems. Personally I prefer tr because it's more readable. But I don't know it's availability outside Unix. You can try single-quoting the sed expression and eliminationg the backslashes altogether: MP_VERSION=`echo $MP_VERSION | sed 's///g' ` If there is also some problem with a sed treating as a special character, you can always use character classes: MP_VERSION=`echo $MP_VERSION | sed 's/[]//g' ` I generally find that avoid backslashes at all cost improves portability of scripts across broken shells/seds. -- Deron Meranda
Enhancements for better content negotiation
I have a bunch of code I was thinking of contributing to mod_python,
but would like some opinions before doing so (because I don't
know if this is the best place)...
Basically I wrote some utility functions which can be used to
assist with content negotiation; such as parsing the various
Accept-* headers *correctly*. I am in fact using these in production
systems now because I do quite a bit of content negotiation.
I often see a bunch of example code (or even production code)
which does something like,
if req.headers_in['Accept'].find('image/xyz') = 0:
That is really broken per RFC 2616 (HTTP 1.1)! It totally ignores the
quality factors and other rules (see sections 14.1 - 14.4). For example
if the Accept header sent was image/xyz;q=0, then in fact this
is the user agent saying to NEVER send image/xyz!
I think this sloppy way of testing is because the syntax and rules
for accept headers is actually fairly complex. So this is a perfect
opportunity for reusable functions which assist with this (much like
the apache.parse_qs function).
Essentially I have functions that:
1. Parse and sort any Accept-* header according to the RFC's BNF
2. Negotiate the best content type
3. Negotiate the best language
4. Negotiate the best encoding
5. Negotiate the best charset
It also knows how to handle wildcards, hierarchical language tags,
charset name aliases, etc. And you can tell it to ignore the super
wildcards like */* too.
As a quick example you can do something like
acc = req.headers_in['Accept']
ct = acceptable_content_type( acc, ['text/html','application/xhtml+xml'] )
and it will tell you which of the two formats the browser supports/prefers,
if either, according to all the complex rules in the RFC.
I have some preliminary questions:
* Is this something that seems useful to others?
* Is mod_python util a preferable place to consider putting these,
or maybe this should perhaps go to something larger like
WSGI or some other Web-SIG project?
--
Deron Meranda
Re: 3.2.6 test period - how long do we wait?
+1 Release what's fixed already, and then keep going afterwards. But can somebody address MODPYTHON-53 please. (updating modpython.org website). -- Deron Meranda
Re: mod_python as a mod_dav backend
Actually it seems that this is yet another case of trying to get mod_python to hook into more places in the Apache framework; specifically to hook into other modules. We've already been discussing specific-module hooks for mod_ssl - http://issues.apache.org/jira/browse/MODPYTHON-94 mod_include - http://issues.apache.org/jira/browse/MODPYTHON-104 In fact there's a bit of potentially useful discussion in the notes for those JIRA issues which is valid for any module-hooking in general (along with this mailing list archives). I'd say it might be worthwhile to at least create a new feature-request issue for mod_dav hooks though; just so it's on the record. BTW, even though it's not what you asked, it should be possible to totally implement DAV using only the features already present in mod_python (without mod_dav); but of course you're doing a lot of the work yourself (but on the other hand you get to use Python rather than C). -- Deron Meranda
Re: 3.2.6 test period - how long do we wait?
On 1/31/06, Jim Gallacher [EMAIL PROTECTED] wrote: apache 2.2. I scanned mod_python for deprecated or removed apr calls and can find only one (apr_sockaddr_port_get), plus the missing APR_STATUS_IS_SUCCESS macro. The apr_sockaddr_port_get() call was introduced by me to support IPv6 in modpython with MODPYTHON-64. Looking at the APR 1.0 CHANGES file (accompaning httpd 2.2): *) The following deprecated interfaces have been removed: apr_sockaddr_port_get- (access directly) I'm trying to review the APR source to see exactly what that means and the conditions under which it's safe. We don't want to accidentally pull the wrong field and have endianness problems. BTW, I want to get this one fixed before release rather than reverting to the pre-MODPYTHON-64 code...or we'll break under IPv6 again. -- Deron Meranda
[jira] Commented: (MODPYTHON-78) No support for Apache 2.2 yet
[ http://issues.apache.org/jira/browse/MODPYTHON-78?page=comments#action_12364671 ] Deron Meranda commented on MODPYTHON-78: One small incompatibility is the call to the APR function apr_sockaddr_port_get(), as introduced in the fix for bug MODPYTHON-64. That function was deprecated in APR 1.0 (which maked it into httpd 2.2). The changelog for APR says: *) The following deprecated interfaces have been removed: apr_sockaddr_port_get- (access directly) After reviewing the APR changes, it appears that the correct fix to mod_python is quite simple. The line in connobject.c: apr_sockaddr_port_get(port, addr); should be replaced with port = addr-port; This is still portable and has no endianness or word-size issues. (as it would if it tried accessing the addr-sa sub-struct directly). No support for Apache 2.2 yet - Key: MODPYTHON-78 URL: http://issues.apache.org/jira/browse/MODPYTHON-78 Project: mod_python Type: Bug Components: core Versions: 3.2 Reporter: Nicolas Lehuen Fix For: 3.3 See http://article.gmane.org/gmane.comp.apache.mod-python.devel/1425 for some remarks by Jorey Bump, raised during the 3.2.1-BETA tests. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: 3.2.6 test period - how long do we wait?
On 1/31/06, Deron Meranda [EMAIL PROTECTED] wrote: On 1/31/06, Jim Gallacher [EMAIL PROTECTED] wrote: apache 2.2. I scanned mod_python for deprecated or removed apr calls and can find only one (apr_sockaddr_port_get), plus the missing APR_STATUS_IS_SUCCESS macro. The apr_sockaddr_port_get() call was introduced by me to support IPv6 in modpython with MODPYTHON-64. Okay, the correct fix for apr_sockaddr_port_get() which works under both Apache 2.0 and 2.2 is to replace the single call in connobject.c, apr_sockaddr_port_get(port, addr); with port = addr-port; I've added this information to MODPYTHON-78 notes (httpd 2.2 port). Of course that fix only *needs* to be made when 2.2 is completely addressed, but it can be safely rolled in before then too. -- Deron Meranda
Re: Server side includes and Python.
I like the SSI feature. It would fill a nice gap between using plain HTML files and having to go to a more featured template or engine. Some things are simple enough that the SSI concept should be enough, and having Python would be nice. I do need to give your proposal some more thought before I can properly comment, but it looks interesting so far. One thing I think that *should* be very easy to do in an SSI setting is HTML-escaping. I shouldn't have to do something like 'from cgi import escpe'. Perhaps adding another parameter, like !--#python esc=h eval=print '12'-- where esc is a built-in escaping filter: h=html, u=url, x=xml (difference between h and x is how it escapes quote chars). Another question. How are character sets handled? If the output is a Unicode string, how does it get encoded? Should it always asume say UTF-8, or can it determine the actual character encoding for this reponse somehow? Also, my vote is that None should result in no output. -- Deron Meranda
[jira] Commented: (MODPYTHON-94) Calling APR optional functions provided by mod_ssl
[ http://issues.apache.org/jira/browse/MODPYTHON-94?page=comments#action_12362005 ] Deron Meranda commented on MODPYTHON-94: Keeping the same mod_ssl name for this function is fine with me. Calling APR optional functions provided by mod_ssl -- Key: MODPYTHON-94 URL: http://issues.apache.org/jira/browse/MODPYTHON-94 Project: mod_python Type: New Feature Components: core Environment: Apache 2 Reporter: Deron Meranda Fix For: 3.3 Attachments: modpython4.tex.patch, requestobject.c.patch mod_python is not able to invoke APR Optional Functions. There are some cases however where this could be of great benifit. For example, consider writing an authentication or authorization handler which needs to determine SSL properties (even if to just answer the simple question: is the connection SSL encrypted). The normal way of looking in the subprocess_env for SSL_* variables does not work in those early handler phases because those variables are not set until the fixup phase. The mod_ssl module though does provide both a ssl_is_https() and ssl_var_lookup() optional functions which can be used in earlier phases. For example look at how mod_rewrite calls those; using the APR_DECLARE_OPTIONAL_FN and APR_RETRIEVE_OPTIONAL_FN macros. I can see how it might be very hard to support optional functions in general because of the C type linkage issue, but perhaps a select few could be coded directly into mod_python. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-94) Calling APR optional functions provided by mod_ssl
[ http://issues.apache.org/jira/browse/MODPYTHON-94?page=all ] Deron Meranda updated MODPYTHON-94: --- Attachment: modpython4.tex.patch This is a documentation patch which goes with the previously attached code patch. Made against 3.2.5b. Calling APR optional functions provided by mod_ssl -- Key: MODPYTHON-94 URL: http://issues.apache.org/jira/browse/MODPYTHON-94 Project: mod_python Type: New Feature Components: core Versions: 3.2 Environment: Apache 2 Reporter: Deron Meranda Attachments: modpython4.tex.patch, requestobject.c.patch mod_python is not able to invoke APR Optional Functions. There are some cases however where this could be of great benifit. For example, consider writing an authentication or authorization handler which needs to determine SSL properties (even if to just answer the simple question: is the connection SSL encrypted). The normal way of looking in the subprocess_env for SSL_* variables does not work in those early handler phases because those variables are not set until the fixup phase. The mod_ssl module though does provide both a ssl_is_https() and ssl_var_lookup() optional functions which can be used in earlier phases. For example look at how mod_rewrite calls those; using the APR_DECLARE_OPTIONAL_FN and APR_RETRIEVE_OPTIONAL_FN macros. I can see how it might be very hard to support optional functions in general because of the C type linkage issue, but perhaps a select few could be coded directly into mod_python. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
